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ABSTRACT 


The Navy and Marine Corps use the ESSENCE system for early detection of diseases and 
other public health threats to the force and for situational awareness on the location and 
spread of such diseases. In accordance with BUMEDINST 6220.12B, the NMCPHC 
sponsored a survey to better understand ESSENCE account holders’ training on the 


system, employment of the system, and their perceived value of the system. 


The survey was sent to 225 Navy and Marine Crops users with either an active or 
a disabled ESSENCE account. Ultimately, 143 of the users responded to the survey for a 


64 percent response rate. 


Survey findings conclude that, overall, nine out of 10 ESSENCE account holders, 
past and present, favor using the system, find it valuable, and believe the training they 
received has been adequate. However, users raised four issues: 1) it takes an excessive 
amount of time to obtain an account, 2) passwords are required to be changed too often, 
3) there are too many miscodings, leading to excessive false positive signals, and 4) 


training and training tools are insufficient. 
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EXECUTIVE SUMMARY 


The Navy and Marine Corps use the ESSENCE system for early event detection 
of diseases and other public health threats to the United States military and for situational 
awareness on the location and spread of such diseases. BUMEDINST 6220.12B 
mandated that “health surveillance shall be conducted to enable early intervention and 
control strategies.” Navy ESSENCE monitors are health professionals, including 
preventive medicine technicians, doctors, nurses, public health professionals, and 


epidemiologists. 


In accordance with the Navy Bureau of Medicine and Surgery (BUMED) 
mandate, the Navy and Marine Corps Public Health Center (NMCPHC) sponsored a 
survey of Navy and Marine Corps ESSENCE account holders about their training, 
employment, and perceived value of the system. This survey differs from previous 
ESSENCE surveys in a number of important ways. First, it is a survey only of Navy and 
Marine Corps ESSENCE users. Hence, it 1s targeted at only those users of interest to 
NMCPHC, and it focuses on their unique Navy/Marine Corps perspectives and issues. 
Second, the survey was sent to all existing and known former Navy and Marine Corps 
users. It targeted all users, not just those who have the resources and ability to attend 
professional conferences (such as International Society of Disease Surveillance (ISDS)). 
Thus, it was intended to garner feedback from the full range of Navy and Marine Corps 
ESSENCE account holders. Third, it focused on areas of particular interest to NMCPHC: 
how the Navy and Marine Corps monitors used the ESSENCE system, what their 


perceived value of the system 1s, and what training they would like to see in the future. 


The survey was sent to 225 Navy and Marine Crops users with either an active or 
a disabled ESSENCE account. Ultimately, 143 of the users responded to the survey for a 
64 percent response rate. Survey findings include that, overall, nine out of 10 ESSENCE 
account holders, past and present, favor using the system, find it valuable, and believe the 


training they received has been adequate. However, users raised four issues: 1) it takes 


xi 


an excessive amount of time to obtain an account, 2) passwords are required to be 
changed too often, 3) there are too many miscodings, leading to excessive false positive 


signals, and 4) training and training tools are insufficient. 
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I. BIOSURVEILLANCE AND ESSENCE 


After the attacks on 9/11, the United States (U.S.) government sought out ways to 
protect the nation from future terrorist attacks. One form of a potentially very deadly 
attack 1s bioterrorism. These attacks can be in the form of biological weapons, perhaps 
consisting of an infectious disease or diseases dispersed throughout the country, with the 
goal of infecting the population and killing members of our society. Examples of 
bioterrorism agents include anthrax, botulism, and epsilon toxin, to name a few. Details 
of these agents and other virulent naturally occurring diseases are listed on the Centers 
for Disease Control and Prevention’s (CDC) Website: http://www.bt.cdc.gov/agent/ 
agentlist.asp. To help protect the country against, and mitigate the effects of, a 


bioterrorist attack, federal, state, and local agencies conduct biosurveillance. 


A. WHAT IS BIOSURVEILLANCE? 


In October 2007, President George W. Bush issued Homeland Security 
Presidential Directive 21 (HSPD-21, 2007). It defines biosurveillance as 

the process of active data-gathering with appropriate analysis and 

interpretation of biosphere data that might relate to disease activity and 

threats to human or animal health—whether infectious, toxic, metabolic, 

or otherwise, and regardless of intentional or natural origin—in order to 


achieve early warning of health threats, early detection of health events, 
and overall situational awareness of disease activity. (HSPD-21, 2007) 


One part of biosurveillance, syndromic surveillance, “is the surveillance of health- 
related data that precedes diagnosis to detect a disease outbreak or other health related 
event that warrants a public response” (Kleiner et al., 2008). Under the umbrella of 
biosurveillance, syndromic surveillance is intended to provide early event detection of an 
illness or attack before any particular individual has actually been diagnosed with the 
specific illness or attack agent. It is also intended to provide situational awareness of the 
location and spread of the illness or agent so that public health and medical professionals 


can localize and treat the affected population or community. 


In 2003, agencies, such as the CDC and state and local health departments, were 
estimated to have approximately 100 sites throughout the country that utilize a syndromic 
surveillance system (Buehler et al., 2003, p. 1197). 

The goal of these systems is to enable earlier detection of epidemics and a 

more timely public health response, hours or days before disease clusters 

are recognized clinically, or before specific diagnoses are made and 

reported to public health authorities. (Buehler et al., 2003, p. 1197) 

In order for early event detection to occur, a syndromic surveillance or 
biosurveillance systems had to be created. On the other hand, a year later, Dena Bravata, 
MD and other authors synthesized numerous articles and concluded, 

Few surveillance systems have been specifically designed for collecting 

and analyzing data for the early detection of a bioterrorist event. Because 

current evaluations of surveillance systems for detecting bioterrorism and 

emerging infections are insufficient to characterize the timeliness or 
sensitivity and specificity, clinical and public health decision making 

based on these systems may be compromised. (Bravata et al., 2004, p. 

917) 

However, in 2008, Buehler and his team conducted a survey of public health 


practitioners in which over two-thirds of the survey respondents indicated that they would 


expand their use of syndromic surveillance within the next two years (Buehler, 2008). 


B. THE EARLY NOTIFICATION OF COMMUNITY-BASED EPIDEMICS 
(ESSENCE) SYSTEM 


A number of systems have been produced that are used at the federal, state and 
local levels. These include the Early Aberration Reporting System (EARS), BioSense, 
The National Electronic Disease Surveillance System (NEDSS) and the Electronic 
Surveillance System for the Early Notification of Community-Based Epidemics 
(ESSENCE). Various federal, state and local health departments operate and monitor 
these systems; however, ESSENCE 1s the only system used by the Department of 
Defense (DoD). 


EARS: “Various city, county, and state public health officials in the 
United States and abroad currently use EARS on syndromic data from 
emergency departments, reportable conditions, 911 calls, physician office 
data, school and business absenteeism, and over-the-counter drug sales. 
EARS is convenient, easy to use, and available at no cost.” (CDCb, 2010) 


BioSense: “In FY 2010, CDC started redesigning the BioSense program 
based on input and guidance from our local, state, and federal partners. 
The goal of the redesign effort 1s to be able to provide nationwide and 
regional situational awareness for all-hazard health-related threats (beyond 
bioterrorism) and to support national, state, and local responses to those 
threats. By integrating local and state-level information, CDC will 
provide a timely and cohesive picture at the regional (i.e., multistate) and 
national levels and improve BioSense's utility.” (CDCa, 2010) 


NEDSS: “is an initiative that promotes the use of data and information 
system standards to advance the development of efficient, integrated, and 
interoperable surveillance systems at federal, state and local levels. The 
vision of NEDSS is to have integrated surveillance systems that can 
transfer appropriate public health, laboratory, and clinical data efficiently 
and securely over the Internet. NEDSS will revolutionize public health by 
gathering and analyzing information quickly and accurately. This will help 
to improve the nation's ability to identify and track emerging infectious 
diseases and potential bioterrorism attacks, as well as to investigate 
outbreaks and monitor disease trends.” (CDC) 


1. The History of ESSENCE 


On June 12, 1996, the Clinton Administration released a fact sheet via the White 
House Office of Science and Technology (OST) stating that 


the National Science and Technology Council [had] determined that the 
national and international system of infectious disease surveillance, 
prevention, and response is inadequate to protect the health of U.S. 
citizens. (OST Policy, 1996) 


As described in the fact sheet, the administration's goals were to: 


I “Strengthen the domestic infectious disease surveillance and 
response system, both at the Federal, State, and local levels and at 
ports of entry into the United States, in cooperation with the 
private sector and with public health and medical communities,” 


2. “Establish a global infectious disease surveillance and response 
system, based on regional hubs and linked by modern 
communications,” and 


> “Expand missions and establish the authority of relevant United 
States Government agencies to contribute to a worldwide 
infectious disease surveillance, prevention, and response network.” 
(OST Policy, 1996) 


It further stated that the United States Government (USG) roles and 
responsibilities were to “enhance the surveillance and response components of our 
domestic and international public health infrastructure [by] strengthen[ing the] federal 
and state laboratory and epidemiological response capabilities. The CDC will coordinate 
federal government efforts to strengthen federal, state and local health department|s’ | 


surveillance and response capabilities” (OST Policy, 1996). 


Following OST's direction, the Department of Defense created the DoD Global 
Emerging Infections Surveillance and Response System (DoD-GEIS) in 1997. With a 
focus on global health surveillance, the DoD-GEIS began with laboratories in countries, 
such as Peru, Indonesia, Egypt, Thailand, and Kenya (Sueker, 2010). A sponsored 
program of the DoD-GEIS oversees the Global Influenza Surveillance Program and 
ESSENCE syndromic surveillance system. “ESSENCE had been [a] developmental 
program for syndromic surveillance using data from the National Capital Area, but 
following the events of September 11, 2001, ESSENCE began receiving data from all 
military medical facilities in DoD” (MacIntosh, 2004). Thus, the first implementation of 
ESSENCE, ESSENCE I, gathered information from “military personnel and _ their 
dependents at all military treatment facilities by using ambulatory records generated for 


TriCare, the military's health-care system” (Lombardo, 2004). 


The next variant, “ESSENCE II was a regional system that supported advanced 
surveillance within the National Capital Region (NCR) test bed. The system [was] 
developed by the Johns Hopkins University Applied Physics Laboratory (JHU/APL) in 
collaboration with the Maryland Department of Health and Mental Hygiene, the District 
of Columbia Department of Health, and the Virginia Department of Health” (Lombardo, 
2004). The DoD-GEIS and JHU/APL also collaborated on creating and updating 
subsequent variants of the ESSENCE syndromic surveillance system. Primarily, their 
work focused on the algorithms necessary for the system to detect syndromic groupings 


before an epidemic has matured into a health catastrophe. What the developers of the 
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system accomplished was to allow “ESSENCE [to] capture data in multiple formats, 
parses text strings into syndrome groupings, and applies multiple temporal and spatio- 
temporal outbreak-detection algorithms. [In a] DARPA evaluation exercise, ESSENCE 
algorithms detected a set of health events with a median delay of | day after the earliest 


possible detection opportunity” (Lombardo, 2004). 


The newest installment of the syndromic surveillance system, ESSENCE IV, 1s 


currently under test evaluations and 1s expected to come online within the next few years. 


Z. Why Does the DoD and the U.S. Navy Use ESSENCE? 


Although civilian health departments use biosurveillance systems, the DoD has 
also made their use mandatory as well. In 1997, DoD Directive 6490.2—Joint Medical 
Surveillance, “establishes policy and assigns responsibility for routine joint medical 
surveillance of all Military Service members during active Federal service, especially 
military deployments.” Then, in 2004, DoD Directive 62004—Force Health Protection 
(FHP), “assigns responsibility for implementing FHP measures, on behalf of all Military 
Service members during active and Reserve military service, encompassing the full 
spectrum of mission, responsibilities, and actions of the DoD Components in 
establishing, sustaining, restoring, and improving the health of their forces.” The August 
24, 2009, DoD Directive 6490.2E incorporating change 1, originally written in October 
2004, states that “the DoD components shall conduct comprehensive, continuous, and 
consistent military health surveillance to implement early intervention and control 
strategies, using joint technologies, practices, and procedures in a manner consistent 
across the military services. Relevant, timely, actionable, comprehensive health 
surveillance information shall be collected and maintained to support the Armed Forces.” 
Following this guidance in 2009, the U.S. Navy and the Navy Bureau of Medicine and 
Surgery (BUMED) mandated that the “monitoring of ESSENCE alerts applies to all 
Navy medical treatment facilities (MTFs). Health surveillance shall be conducted to 


enable early intervention and control strategies. Monitoring for and timely reporting of 


significant health events that may adversely affect mission accomplishment and shape the 
commander’s decision making is vital military medicine functions” (BUMEDINST 


6220.12B 2009). 


3. How Does the ESSENCE System Work and Who Uses It? 


The military treatments facilities that use the ESSENCE system capture data from 
the military health system (MHS), which includes outpatient clinical visits, pharmacy 
transactions, and laboratory orders. The ESSENCE system then organizes the ICD-9 
(International Classification of Diseases 9" Edition) codes! input by the doctor, nurse, 
laboratory technician, or other health professionals, and analyzes and sorts the data into 
syndromic types that an operator of the system can view. Via the use of statistical 
algorithms, the system seeks to identify emerging or re-emerging infectious diseases that 
pose a substantial risk to a commander’s mission or the U.S. Navy’s mission. The 
algorithms are intended to allow the user to see increases and decreases in a particular 


syndrome so they can monitor its activity in their region. 


Again, the ESSENCE system is used for Early Event Detection (EED) and 
location situational awareness (SA). The CDC definition of EED is “the ability to detect, 
at the earliest possible time, events that may signal a public health emergency. EED 1s 
comprised of case and suspect case reporting along with statistical analysis of health- 
related data. Both real-time streaming of data from clinical care facilities, as well as 
batched data with a short time delay are used to support EED efforts” (CDC, 2008). The 
definition of situational awareness 1s “the ability to utilize detailed, real-time health data 
to confirm, refute and to provide an effective response to the existence of an outbreak. It 


also is used to monitor an outbreaks magnitude, geography, rate of change and life cycle” 


(CDC, 2008). 


Although these terms are similar in definition, they actually provide the monitor 


with two mechanisms towards understanding how and why the ESSENCE program was 


! ICD-9 clinical modification codes are defined as “a standardized classification of disease, injuries, 
and causes of death, by etiology and anatomic localization and codified into a 6-digit number, which allows 
clinicians, statisticians, politicians, health planners and others to speak a common language, both US and 
internationally” (The Free Dictionary, 2010). 
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developed and how to use it properly. EED allows the user to find communicable 
diseases that are on the rise in their area prior to a doctor diagnosing the event. Because 
an account holder is able to monitor multiple military treatment facilities at one time, it 1s 
possible to observe increases using tools, such as a time-series graph (which will be 
explained below) and alert status received from the ESSENCE system itself. Situational 
awareness, on the other hand, gives the monitor the ability to observe multiple MTFs and 
to understand which already diagnosed illnesses are on the rise and continue to rise, as 
well as be able to monitor the patients that enter the MTFs via city, state, region, and 


country. 


The Navy ESSENCE monitors are health professionals, including preventive 
medicine technicians, doctors, nurses, public health professionals, and epidemiologists. 
The agencies that monitor ESSENCE are the preventive medicine and public health 
departments of the Navy throughout the world. They are comprised of naval clinics, 
naval hospitals, and naval medical centers. The Navy Environment Preventive Medicine 
Units (NEPMU), which are located in San Diego, California, Pearl Harbor, Hawau, and 
Norfolk, Virginia, also monitor disease activity using ESSENCE. Finally, the Navy and 
Marine Corps Public Health Center (NMCPHC) oversees patient syndromic surveillance 
throughout the Navy as mandated by the BUMEDINST 6220.12B. 


C. WHY SURVEY ESSENCE ACCOUNT HOLDERS? 


In accordance with their BUMED mandate, the NMCPHC wants to better 
understand how ESSENCE is being used by surveying Navy and Marine Corps 
ESSENCE account holders about their training, employment, and perceived value of the 
system. For example, NMCPHC has noticed that some installations that are supposed to 
be viewing and using the ESSENCE system on a daily basis are not. Thus, they want to 
understand the reason why this phenomenon is occurring. In addition, the NMCPHC has 
little information about whether their users understand how to use the system effectively 


and efficiently or whether users think the system is providing them with the proper 


information they need. Furthermore, the NMCPHC is interested in monitors' training and 


usage habits of biosurveillance and/or syndromic surveillance. 


ie Related Research 


In 2007—2008, the International Society for Disease Surveillance (ISDS) and the 
CDC sponsored a survey of 59 state, territorial, and selected large local jurisdictions in 
the United States regarding their use of syndromic surveillance. The survey defined 


syndromic surveillance as systems with all of the following characteristics: 


e surveillance for human health-related events or outcomes; 

e surveillance for the purpose of early event detection or situational 
awareness; 

e ongoing surveillance as opposed to time-limited, ‘drop-in’ surveillance 


around specific high-profile events; and, 


® surveillance systems not established primarily to support notifiable disease 
reporting (Buehler et al., 2008). 


The ISDS survey was e-mailed to respondents and telephone notifications made. 
A remarkably high response rate of 88 percent was achieved, and the responding 
jurisdictions accounted for 94 percent of the U.S. population. Eighty-three percent of 
respondents “reported conducting syndromic surveillance for a median of 3.3 years. 
Emergency data visits were [a] commonly reported information source (84 percent). 
Seventy-two percent reported reviewing the data from participating hospitals at least 
daily” (Buehler, 2008). Ninety-three percent reported that syndromic surveillance was 


“highly useful or somewhat useful” for “monitoring influenza” (Buehler, 2008). 


Moreover, syndromic surveillance was concluded to be more useful when 
tracking and ILI and other larger communicable diseases rather than smaller or rarer 
diseases. Over two-thirds of the respondents indicated that they would expand their use 
of syndromic surveillance within the next two years (Buehler, 2008). Buehler and his 
team of researchers deduced that syndromic surveillance would continue to expand and 
that it 1s important to provide future users training and lessons learned on how to monitor 


their systems correctly. 


The Johns Hopkins Applied Physics Laboratory (JHU/APL) conducted another 
survey in 2009 on behalf of the Armed Forces Health Surveillance Center (AFHSC). 
Their objective was to “examine the strengths and weaknesses of the current ESSENCE 
system, by capturing feedback from current DoD ESSENCE users” (JHU/APL, 2009). 
JHU/APL developed a survey, which they entitled the “DoD ESSENCE User Feedback 
Tool” and the AFHSC sponsored a workshop during the 2009 Force Health Protection 
Conference. Thirty-one DoD ESSENCE users participated in the workshop and 17 DoD 
ESSENCE users responded to their survey. Although the number of monitors was low, 
they were able to obtain worthy and significant feedback because those who did 
participate voiced their concerns honestly and openly (JHU/APL, 2009). JHU/APL 
concluded that the most important issue for ESSENCE is a “need for user training both in 
how and when to use the functions available in the tool, as well as how to interpret the 
resulting system displays.” They further concluded that the “second most raised issue, 
with regard to the currently deployed system, was the impact that incorrectly coded 
information, as well as delays in the coding information has on both the timeless and 
usefulness of the information displayed and the resulting alters generated by the system” 


(JHU/APL, 2009). 


Zz. How Is This Survey Different? 


This survey, conducted on behalf of the NMCPHC, differs from the JHU/APL 
survey in a number of important ways. First, only Navy and Marine Corps ESSENCE 
users were surveyed. Hence, it targeted only those users of interest to NMCPHC and 
focused on their unique Navy/Marine Corps perspectives and issues. Second, the survey 
was sent to all existing and known former Navy and Marine Corps users, 225 in all. 
Therefore, it targeted all users, and not just those with the resources and ability to attend 
professional conferences, such as ISDS. Thus, it was intended to garner feedback from 
the full range of Navy and Marine Corps ESSENCE users. Third, it focused on areas of 
particular interest to NMCPHC: how the Navy and Marine Corps monitors used the 
ESSENCE system, what their perceived value of the system was, and what training they 


would like to see in the future. 


D. ORGANIZATION OF THIS THESIS 


This thesis in organized as follows. Chapter II describes the background research 
and how the survey instrument was designed. Chapter III describes the survey approval 
process, how the survey was pretested and fielded, and the survey response rate. Chapter 
IV presents the survey results. Finally, Chapter V summarizes the findings and provides 
recommendations for steps the NMCPHC can take towards improving the ESSENCE 


system. 
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I. SURVEY DESIGN 


This chapter discusses my background research and how the survey instrument 


was designed. See Appendix A for a copy of the survey instrument. 


A. BACKGROUND RESEARCH 


To understand fully how monitors utilize the system, I traveled to several MTFs 
around the country. In Portsmouth, Virginia, I spoke to the (former) ESSENCE Program 
Officer, who demonstrated the functionalities of the program, such as the alert list, the 
reportable and syndromic queries, site selections, and the alert e-mails. Moreover, 
instruction was given on how a monitor should use the system effectively to define 
syndromes to detect a particular disease early in its maturity and also for the situational 
awareness of a disease as well. She also identified how miscoded ICD-9’s force the 
monitor to investigate outside sources, such as the Composite Health Care System 
(CHCS) and Armed Forces Health Longitudinal Technology Application (AHLTA), to 
discern the proper identifying code for the patient and assess if it contributes to an 


increase in a communicable disease in a specific area. 


Following the visit to Virginia, I created a questionnaire to ask the monitors 
particular questions during future visits. The questionnaire was comprised of desired 
responses from my sponsor and questions I knew would help me understand what a 


monitor does on a daily basis and what can be modified for future ESSENCE use. 


I then met with monitors at NEPMU-5, the Naval Hospital (NH) Camp Pendleton, 
and the Naval Medical Center (NMC) San Diego. First, I began by speaking with the 
regional monitor at NEPMU-5 and learned that regional monitors use a wider site 
selection criterion to track the communicable diseases on the rise in their region. The 
regional monitor stated that the role of a regional monitor is different from local 
monitors. She does not investigate alerts; however, she does oversee that her region’s 
monitors are accurately investigating alerts and reporting outbreaks up their chain of 


command. 
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Next, I visited local monitors from the NH Camp Pendleton and from the NMC 
San Diego. My last visit was to Jacksonville, Florida, where I met with my sponsor, Mrs. 
Asha Riegodedios at the NMCPHC. Before the meeting, I spoke on the phone with two 
local monitors from Naval Air Station Pensacola, Florida. The local MTF users provided 
similar types of feedback. They also solidified my understanding of how a local sponsor 
works ESSENCE into their daily routine and what drives them to investigate alerts and 
the methods used to conduct the investigations. An important note from these monitors is 
that they were only taught a small amount of what biosurveillance is in C-school, and 
thus, had to learn the capabilities and functionalities of the system via hands-on training. 
The one of the most crucial take-aways I received from these users was that they spend a 
lot of time investigating alerts in ESSENCE by using CHCS and AHLTA. They are 
required to confirm that an alert is real and if the trend in that particular communicable 


disease 1s on the rise. 


However, when asked if the various users had ever identified an outbreak by 
monitoring ESSENCE alerts, half said they did and half said they did not. Those who did 
identify outbreaks said that this was the very reason they valued the program. By digging 
through the data details to specify site selections, date intervals, and syndromes, and 
coordinating their efforts tracking patient history in ALTHA, they could track where the 
outbreak originated. The value the local MTF monitors have for ESSENCE also varied. 
A few indicated that it provides excellent situational awareness as a means of detecting 
rises in communicable diseases. ESSENCE provides them an “X” that marks the spot 
and through ALTHA and CHCS, they are able to dig down through the data to discover if 


an actual outbreak is occurring in their community. 


On the other hand, some believed the system has tremendous value, but at times 
alerts appear when they should not, due to either miscodings or routine work done by 
doctors in a particular hospital specializing in a certain disease. Also, the system 
considers the days that the hospital or clinic 1s closed as zero visits that day, which 1s 
inaccurate and drives the running average down over the “14-day period, making it 


problematic as it should be longer for a more accurate baseline.” 
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After speaking with and discussing the system with actual monitors, I concluded 
that there are varied opinions about the utility and value of the ESSENCE program, and 
therefore, these aspects must be looked addressed in the survey. In particular, how many 
ESSENCE monitors have these same opinions and how many monitors hold differing 
opinions? This then became the heart of the research, My meeting with Mrs. 
Riegodedios consisted of discussing these particular comments that the monitors said 
about their use and value of the system, and what changes they would like to make. It 
was then possible to compile a few notes and specific topics that the survey needed to be 


addressed by the survey. 


B. SURVEY INSTRUMENT DESIGN 
1. Specifying the Survey Objective 


In her paper, “Fundamental of Survey Research Methodology, Priscilla Glasow 
writes, “survey instrument development must be preceded by certain prerequisites. First, 
the focus of the study must be carefully defined. Second, the study objectives must be 
translated into measurable factors that contribute to that focus” (Glasow, 2005, pp. 2—5). 
It follows that writing the objective of ones’ survey first helps to outline the layout and 


flow of the survey. 


Therefore, after I had completed the research travel, I had some idea of what 
ESSENCE monitors’ day-to-day experiences were, of what their training might or might 
not be comprised of, and what they believed the ESSENCE system provided them. 
Given this information, and considering the information my sponsor wanted to learn from 
the survey, as well as what I thought the survey should include, I specified three main 
survey foci: usage, value, and training. Thus, the survey objective became the following: 
to assess ESSENCE account holder’s usage of, perceived value of, and training of the 


system. 


2. Designing the Survey Instrument 


Given the objective, I then began drafting the survey questions. Initially many of 


the questions were overly wordy, sometimes less than completely coherent, and the 


13 


question structures and formats were mixed around. For instance, the questions that 
pertained to value were preceded by the following: Do you agree or disagree with the 
following statements regarding the ESSENCE program? This question was subsequently 
modified because it only allowed a binary response and thereby limited the amount of 
information a respondent could convey about the system's value. In the final version of 
the survey, the question read: How much do you agree or disagree with the following 
statements regarding the ESSENCE program? The subtle difference is in the format of 
the question, where the final survey allows the respondent to answer naturally on a 5- 


point Likert scale. 


Given a set of draft questions, Professor Don Dillman, a leading expert on survey 
methodology, says that effort should be “made to order questions in a way that will be 
logical to the respondent” (Dillman et al. 2008, p. 88). In the first design of the survey, 
the question structure was found to be confusing to respondents and did not flow 
properly. Also, the format of the questions that would allow me to assess monitor usage 
was disheveled and lacked a chronological ordering. For instance, questions about time- 
series graphs and queries were hidden within the middle of the survey, rather than 
arranged in the order that respondents found most natural: alert e-mails, queries, 
investigations, and then data details. The use of time-series graphs, in my opinion, 1s 
better asked under the category of perceived value because it has the potential to assist 
the monitor in identifying outbreaks, and thus, providing situational awareness advertised 
by the proponents of syndromic surveillance. However, shifting the questions around so 
they occur in the natural ordering as 1f someone where to log into ESSENCE and conduct 
their daily alert and query check, helped guide the respondent in remembering how they 
use ESSENCE. 


Finally, I gave some thought about how to start the survey. As Professor Dillman 
states, “choose the first question carefully” (Dillman et al., 2008, p. 92). He continues by 


saying that it should be “easy,” “interesting,” and “apply to everyone” (Dillman et al., 


2008, p. 92). The first question does just that: “On average, how often do you log into 
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ESSENCE?” From this first question, the survey then flowed through five major 
categories of questions: monitor usage, perceived value, user training, use of the system, 


and demographics. The following subsections describe each of these question categories. 


a. Assessing Monitor Usage 


The background research I gained through traveling and speaking with 
different monitors throughout the country provided insight as to how this section of the 
survey must be developed. Understanding their log-in habits and module usage 
contributed towards assessing monitor usage. For instance, the survey begins with the 
following question: “On average, how often do you log into ESSENCE?” This first 
question draws the participant into the survey, it 1s easy to answer, and it begins with 
logging into the program. The next set of questions asks the monitors whether they 
receive e-mail alerts from ESSENCE and if so, if that 1s the reason they are logging in. 
The next set of questions ask monitors if they use the alert list, reportable diseases, and 
syndromic query modules and what they functionalities they use and why they perform 
them while conducting surveillance and SA in those modules. In the last part of this 
section, questions were asked to gauge which users investigate alerts and if so, how many 
use ESSENCE to investigate those alerts and what site selections, data details, external 
tools, and recording methods do they use. Next, in this section, the survey strives to elicit 
how monitors use the different modules to view disease activity. The three different 


modules are the alert list, the reportable diseases query, and the syndromic query. 


The goal of this section is to help the NMCPHC understand the usage 
patterns of the monitor and ascertain how they utilize the different functionalities of the 
system during their logged in time. Since the alert identifies what syndromic diseases 
have increased in a particular area, or site selection, then 1t can be understood if the user 
wants to see the general disease count in that area. The alert list is perfect for EED 
queries within a health department. A syndromic query allows the user to view specific 
diseases within a certain date range and location. By specifying a particular syndrome, 
for instance the influenza-like-illness (ILI) syndrome, the monitor can visually look at the 


syndrome’s increased and decreased daily count over time for situational awareness 
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purposes. If the commander of a Naval Medical Center wanted to know when to begin 
administering flu shots, the preventive medicine technician would need to download the 
time-series table for the syndrome for their hospital and look at the when the flu 
increased in outpatient care to give with certainty when the next epidemic of the flu 
might occur this year. Using past history is a useful way for any MTF to gain SA within 
their area. Lastly, the reportable diseases query allows the Navy monitor to collaborate 
with their fellow Navy Disease Reporting System Internet (NDRSi)* account holder. 
This allows the two individuals to confirm that the 24-hour reportable diseases, as well as 
the diseases that must be reported within 30 days, have been sent up their chain of 


command via the NDRSi system. 


Finally, comprehending who investigates alerts provided the data essential 
to understanding which monitors’ spend time navigating through the program and 
assessing true outbreak detection or false positives. The question that needs to be asked 
here is if ESSENCE 1s used to provide their chain of command with outbreak information 


or EED information and SA. 


It is possible to correlate many of these questions with the demographics 
of the participants to understand which rank and professional background/position access 
a particular module. For instance: do preventive medicine technicians, who are on 
average mid-grade enlisted personnel, regularly use the program compared to, say, a 
high-ranking officer, such as the commander of the preventive medicine? Furthermore, 
answers to questions, such as are users with advanced training through the use of 
ESSENCE rather than those with graduate degrees understand the nuances of the system 
better and comprehend the backbone of the system better? It 1s also important to discover 
what the daily users would like to see changed within the program so that their time spent 


logged into ESSENCE 1s efficient and effective. 


2 As per BUMED INST 6220.12B, Medical Event Reporting (MERs) are “required for select diseases 
and injuries due to their potential to compromise operational readiness, present hazards to the military or 
civilian community, result in quarantine, or generate inquires to the Chief, BUMED.” The preferred 
method for MER submittal is by using the Web-based NDRSi system. 
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b. Assessing Perceived Value 


Respondents' perceived value of the ESSENCE system is difficult to 
assess via questions in a survey. If the surveyor asks the respondents outright if they 
value the system, the response may be different than asking them an array of questions 
that represent what the system should be doing for the user and if it is providing them 
with the results for which it was originally intended. As mentioned before, ESSENCE 
account holders’ opinions are varied, and their value in the system to identify outbreaks 


correctly and provide situational awareness is important to the NUCPHC. 


The first iteration of the survey asked whether the account holders value 
the system and the format of this section was not appropriately positioned. After the final 
iteration of the survey, the questions were reformatted to explore the respondents’ 
understating of outbreak detection while using ESSENCE, how they handle miscodings, 
their opinions of the system itself, which were derived from the NMCPHC’s Website, 
which displays the strategy and policy of the ESSENCE system: http://www- 
nehc.med.navy.mil/Preventive Medicine/Disease Surveillance/essence.aspx. The user 
guidance can be found on this page as well and it also helped to formulate questions that 
made it possible to understand the opinions of the monitors. Finally, how their 


experiences in obtaining their current ESSECE are considered. 


Thus, in the final survey, a four-question set was written to understand 
user value. The first asked the respondents if they had discovered an actual outbreak 
using ESSENCE and the next question asked which outbreak types they discovered while 
using ESSENCE. The next question set asked if monitors observed miscoded ICD-9 
codes and what they did to fix them in the future. This was followed by a question that 
asked monitors if they agreed or disagreed with statements regarding the ESSENCE 
program. Finally, the last set of questions asked how long it took the monitors to acquire 
their current ESSENCE account, if they had to apply more than once, and if they had to 
speak with an ESSENCE helpdesk representative while waiting to access their current 


account. 
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The most relevant question set in this entire survey was to ascertain which 
portion of the population discovered an actual outbreak using ESSENCE. The next 
question, on the same page, asked the respondent to check all the outbreak types 
discovered using ESSENCE. After much debate, this question was not skipped if the 
respondents answered no to the first question, but allowed the individual to answer if they 
choose to do so. The interesting question that developed was whether the individual 
would answer no to detecting an actual outbreak and checking the types of outbreaks they 
discovered using the system. If they answered no, but checked outbreaks, did they find 
ghost outbreaks or false positives that were not actually there? Did they not fully 
understand whether the outbreak occurred or not? If account holders were discovering 
false outbreaks, then perhaps they did not fully understand how the system worked or 


was system too sensitive for users. 


Miscodings were put into the perceived value category because if users 
were spending too much time deciphering miscodings, then they were not likely to value 
the system because they were wasting their time eliminating the miscodings from their 
collected information and must deduct the information in their head, rather than having 
ESSENCE remove the miscodings itself. This could also lead to false outbreak 
detections that lower the value of the system and lower the SA and EED of the system as 
well. Perhaps comments pertaining to how respondents would like to correct these 
mistakes might be viable to the longevity of the system by its users and the DoD in 


general. 


The monitors’ opinions of the ESSENCE system are vital to understanding 
how they perceive its value. Thus, the respondents were asked if they agreed or 
disagreed with a set of eight questions with 5-point Likert response scale. If they used 
the system how the NMCPHC perceived them to use it, then their value in the system 
was higher than those who disagreed with the set of questions. These questions were 
reformatted to explore the respondents’ opinions of the system, which were derived from 


the NMCPHC’s Website, which displays the strategy and policy of the ESSENCE system 
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http://www-nehc.med.navy.mil/Preventive_ Medicine/Disease_Surveillance/essence.aspx. 





The user guidance can be found on this page as well and it also helped to formulate 


questions that allowed me to understand the opinions of the monitors. 


Due to the transient nature of the military, 1f monitors go overseas in 
support of the Navy’s public health and preventive medicine missions, and if they 
experience hardships to reacquire their accounts, then their value in the system could be 
diminished and they might not want to waste their busy day trying to access the system. 
The survey assisted in correlating whether the ease of regaining an account after it had 
been deactivated would increase the monitor’s value of the system and their willingness 


to use it. 


In my opinion, this analysis, we expect to correlate which rank and 
professional background/position observed an actual outbreak using ESSENCE and who 
has seen mostly false-positives. If miscodings have been corrected, then monitors should 
see more value in the system in the future. Furthermore, if the system were being used as 
originally intended, then ESSENCE would be valued. However, for those who disagree 
with these statements, who are these monitors and what comments have they made to 


make ESSENCE more effective and efficient to use? 


C. Assessing User Training 


In this last major section of the survey, questions were devised to assess 
user training to understand what has been done in the past and what can be improved 
upon for future users. The first question asked the monitors where they received their 
ESSENCE training. The next set of questions ask where, how, and what the user would 
like to see provided for training in support of current and future users of the ESSENCE 


system. 


The Navy and Marine Corps Public Health Center wants to provide 
ESSENCE users with the best training possible. First, the questions were formatted to 
understand what their past training has been, then to ascertain what they would like to see 
in the future. Knowing where respondents received their training from impacts their 


value of the system and their ability to use the system effectively and efficiently. If the 
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NMCPHC can provide the multiple resources to teach future users, the questions within 
the survey can help direct them towards that goal. Three questions achieve the what, 


where, why and how aspects for future learning tools, modes, and methods. 


I expected to learn that most users had been self-taught on the system and 
they would like to attend conferences and office meetings to learn about ESSENCE and 
its functionalities in particular cases, such as outbreak scenarios and how to evaluate 


alerts properly. 


d. Ease and Effective Use of the ESSENCE System 


This section was not a major category; however, these open-ended 
comment fields allowed the respondent to freely explain what they would change if they 
could and to provide their opinions about what they believed made the system easy to 
use. Also, a final open-ended comment field allowed the monitors to provide any other 
information about the ESSENCE system, particularly topics and issues not asked about in 


the survey, that they felt were important. 


é. Demographics 


It is common to place demographic questions at the end of a survey 
because these questions are likely to be the least important. The goal is to minimize the 
amount and type of information lost should a respondent fail to complete the survey. 
Although not important, as stated previously, correlations can be made throughout the 
major categories and demographics to tie the results together and make them clearer 
during the analysis phase. For example, knowing how many MTFs the user monitors 
allows the NMCPHC to understand how much time can be spent on ESSENCE by the 
monitor and how much training they need in the future. Furthermore, their professional 
backgrounds provide insights into the training the user may have had about syndromic 


surveillance and what it means for DoD, the Navy, and the command. 
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Hil. FIELDING THE SURVEY 


This chapter describes the required survey approvals, how the survey was 


pretested and fielded, and the survey response rate. 


A. OBTAINING THE REQUIRED APPROVALS 


Prior to conducting any survey, approval must be obtained. For surveys within 
the confines of NPS, at a minimum, approval must be obtained from the NPS Institutional 
Review Board (IRB). In addition, since this survey concerns Navy and Marine Corps 
personnel outside of NPS, approval must be obtained by the appropriate survey authority 


within the Navy. 


1. NPS Institutional Review Board Approval 


Before the survey could be administered to the Navy and Marine Corps monitors, 
approval had to be obtained from the NPS IRB. IRB approval is required for any 
research effort involving intervention or interaction with individuals part of a systematic 
investigation and designed to contribute to generalized knowledge. All of these 
conditions applied to this research where, by definition, a survey 1s part of a systematic 
investigation that requires interaction with individuals, and this research is specifically 
intended to result in generalized knowledge about the ESSENCE system in particular and 


biosurveillance in general. 


The purpose of the IRB is to protect individuals involved in any research to 
ensure that the research is conducted in an ethical manner, as well as that all individuals 
participating in the research are fully and correctly informed about the research and 
consent to participate in said research. Per SECNAVINST 300.39D, “the rights, welfare, 
interests, privacy, confidentiality, and safety of human subjects shall be held paramount 
at all times ands all research projects shall be conducted in a manner that avoids all 


unnecessary physical or mental discomfort, and economic, social or cultural harm.” 
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Furthermore, NAVPGSCOLINST 300.4 outlines the necessity to protect volunteers from 
undue burden during an experiment or research. Also, the instruction outlines the 


mandatory documents that must be included in a packet sent to the IRB chair. 


The IRB application requested approval to execute the survey. The application 
describes the study design, including how the data was collected and safeguarded, the 
risks associated with the research, the steps taken to minimize risks, and to protect the 
subject’s welfare and the potential benefits of this research. NPS IRB approval was 
obtained on October 22, 2010. See Appendix B for the IRB approval letter, NMUCPHC 


approval letter, and other IRB documents. 


2. OPNAV N-1 and BUMED Approval 


Since this survey targeted personnel outside of NPS, Navy survey approval was 
also required. OPNAV Instruction 5300.8C—Coordination and Control of Personnel 
Surveys defines the approval process and it “applies to all surveys of the Department of 
the Navy military members. This instruction covers surveys of active duty and reserve 
Navy sailors...civilian employees, as well as those retired from active duty, reserve or 
civilian status.” Like the instructions for the NPS IRB, this instruction protects Navy 
personnel from undue influence and excessive survey burdens. The Navy Survey 
Approval Manager is the Washington, DC liaison, Navy Survey Approval Manager and 
Director of the Institute for Organizational Assessment for the Navy Personnel Research, 
Studies, and Technology Department (NPRST/Pers-14). In most cases, to obtain survey 
approval and receive an OPNAV Report Control Symbol (RCS), the Navy Survey 
Approval Manager must determine that a survey has never been performed before (..e., 
that the proposed survey is not either duplicating a prior effort or that the data being 
sought is available via other sources), that it is endorsed by SES level sponsors, and that 
the survey is being conducted in a technically rigorous manner that maximizes the 


information obtained with minimal respondent burden. 


However, because BUMEDINST 6220.12B states, “NMCPHC shall develop 
methods to improve and facilitate medical surveillance, response, and communication. 


This includes regular program evaluation to identify areas for process improvement,” the 
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Navy Survey Approval Manager determined that only BUMED approval was required 
for this survey. Thus, the approval process was delegated to the Forms and Reports 
Manager for BUMED and the final draft of the survey was submitted to him. “Given 
[that] the BUMED Instruction [supports this survey], I'm okay with considering this 
fulfilling a BUMED requirement rather than a Navy-wide personnel survey” (P. 
Rosenfeld, personal communication, October 6, 2010). Upon completion of the Report 
Analysis Data form, OPNAV 5214/10, signed by CAPT Clagget, the Director of the 
Preventive Medicine Department at the NMCPHC, the survey was assigned RCS 
BUMED 6220-3. To demonstrate that the survey was officially approved, this Report 
Control System number was displayed to recipients of the notification e-mails via the 


subject line and it was displayed on the title line of the survey itself. 


B. PRETESTING THE SURVEY 


Good survey practices dictate that pre-testing a survey helps minimize undue 
respondent error, which can be done in several ways. “Stage 1: Review by 
knowledgeable colleagues and analysts” (Dillman et al., 2008, p. 140). First, I had my 
sponsor, advisor, and the ESSENCE Program Officer review the questions for technical 
merit and question readability. They also looked at whether all the necessary questions 
were included. Since pretesting occurred twice during the design/fielding phases, a more 


suitable and concrete survey was developed as mentioned in the previous chapter. 


In the second stage, “interviews to evaluate cognitive and motivational qualifiers” 
were conducted. I cognitively interviewed two ESSENCE monitors over the phone. 
Two of the interviews were conducted with health professionals, ranging from a medium- 
to-highly ranked enlisted health professional, to a medium-to-highly ranked officer. The 
cognitive interviews allowed me to gauge whether respondents interpreted all the 
questions in the manner intended, both in the most fundamental sense that they 
understood all the words, as well as that they understood the broader intent of the 
question. In addition, cognitive interviews are useful for verifying that “all the questions 
[are] interpreted similarly by respondents” (Dillman et al., 2008, p. 141), and that the 


response scales are appropriate, understandable, and correct. One of the most important 
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results of cognitive interviews 1s to ascertain 1f each respondent can answer each question 
(Dillman et al., 2008, p. 146). As a result of the cognitive interviews, I wrote two 
different surveys because the first iteration contained some confusing answers and did not 


fully represent the complete ESSENCE monitor population. 


The last stage of pretesting involved asking people who had not participated in the 
cognitive interview and revision process to take the survey. A complete fielding mock-up 
the survey was given to them just as it would have been provided to the actual 
respondents. In this final iteration, the survey was pretested by two Navy ESSENCE 
users. In particular, they checked to see if I “did something silly,” as Dillman so 
adequately stated (Dillman et al., 2008, p. 147). When working so closely on a project, 
anyone can loose sight of the goal, and these pretesters provided the sanity check the 


survey needed before it went into the official fielding phase. 


C. WEB-BASED FIELDING 


The survey was administered using SurveyMonkey, a commercial Web-based 
survey software system. The advantages of administering a Web-based survey rather 
than paper-based survey are that it minimizes cost, as well as the ease of distributing 
notification e-mails more than once. Web-based surveys also tend to be faster than 
paper-based survey that must be mailed to respondents and they eliminate data entry. In 
addition, skip logic can be automated within the Web-based survey, which makes it 


easier and faster for a participant to take the survey. 


Due to the transitory nature of the military, the Navy and Marine Corps monitors 
asked to participate in this survey may have transferred jobs or billet locations since the 
last time they registered for an ESSENCE account. Also, since active and disabled 
account holders were included on this list of possible participants, 1t would be difficult to 


track down the location of each member when the survey was administered. 


E-mails are the best way to locate service members regardless of their location. 
Without knowing precisely where monitors could have been working when the survey 
was open would require a multitude of paper surveys sent all over the world to track 


down each monitor. The cost of stamps and envelopes would also increase as well. 
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CAPT Clagett sent out a pre-notification e-mail, discussed below, to notify the 
monitors that the official survey would be sent the next day. If the survey were paper- 
based, then the survey would have had to wait another week until it was first delivered. 
Furthermore, disseminating multiple follow-up letters would have been difficult to 
accomplish since it could have taken weeks before respondents returned their original 
survey. The Web-based method made it possible to send four follow-up e-mails, without 
sending them to personnel who had already responded. The respondents’ answers were 
collected and stored the day they completed the survey, which allowed me to send a 


follow-up e-mail that included responses from a few open-ended questions. 


Lastly, multiple pages of this survey could have been skipped due to the answer 
the respondents gave. If the survey were administered via paper, then it would have 
looked too cumbersome and lengthy for the monitor to complete. Also, the ease of 
having me put the skip logic together, rather than having the participant do it, would have 


led to confusion resulting in more people not finishing the entire survey. 


D. RESPONSE RATES 


The survey was sent to 225 Navy and Marine Corps users with either an active or 
a disabled ESSENCE account. The survey pre-notification e-mail with CAPT Clagget's 
signature block and return e-mail address was sent on November 8, 2010 to all potential 
respondents. The first invitation e-mail was sent out the following day, asking the 
monitors to take the survey. Five follow-up e-mails were subsequently sent to 
nonrespondents to encourage them to take the survey. The last e-mail was sent on 
December 6, 2010. Thus, the survey was in the field for eight weeks. See Appendix C 


for copies of all respondent correspondence. 


A total of 143 monitors (64 percent) responded to the survey, and 7 (0.03 percent) 
opted out of taking the survey. Figure | illustrates the percentage of participants who 
responded per day and the vertical line indicates when the pre-notification, initial 


notification, and follow-up e-mails were sent. 
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The Percentage of Monitors who Responded to 
the Offical ESSENCE User Survey (BUMED 6220-3) 
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Figure 1. | The percentage of respondents who replied to the survey while it was open, 
from November 8, 2010 until December 30, 2010. The bubbled areas 
indicated the dates when pre-notification, notification, and follow-up e- 
mails were sent. The black line shows the response rate over time. 
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IV. SURVEY RESULTS 


This chapter discusses the survey results. Overall, the monitors who responded to 
the survey were generally quite happy with the current ESSENCE system. A few 
individuals suggested how to better the system. Furthermore, most of the respondents 
indicated they use the system as it was intended to be used: for outbreak detection and 
situational awareness. However, a select group is experiencing or has experienced 
problems with the system, all of which are discussed in more detail below. Complete 
survey results, including all open-ended question responses, are contained in Appendix 
D. Appendix D also includes a complete list of survey respondent suggestions for 


making the ESSENCE system easier to use and more effective. 
A. ASSESSING MONITOR USAGE 


The survey begins by asking respondents how often they log into the ESSENCE 
system. As shown in Figure 2, across all types of users, 57 percent log in routinely (that 
is, at least a couple of times per week). On the other hand, 6 percent of the respondents 
said they never log in to ESSENCE. There are no statistically significant differences in 


login frequency by rank, professional background, support facilities, or user level. 


Q2: On average, how often to do you log into ESSENCE? 
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Figure 2. _—_ Distribution of how often respondents logged into ESSENCE. 
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When asked whether they log into ESSENCE only after receiving an e-mail alert, 
roughly one respondent in six (13.4 percent) said yes. As shown in Figure 3, this type of 
log in behavior differs by rank (y° = 9.075, p = 0.0283),3 where 100 percent of the 
contractors said they only logged in after receiving an e-mail alert, followed by 31 
percent of the O4—O6s, 15 percent of all enlisted personnel, and none of the O1—O3s or 
civilians said they logged into the system due to an e-mailed alert. (Caution is warranted 
when interpreting the EI-E3, E7—E9, and a contractor category as the total number of 
respondents in each of these is small.) One possible explanation for this result, which 
cannot be confirmed by the survey data, is that monitors who only log in for e-mail alerts 


have other tasks that prevent them from logging into ESSENCE more frequently. 


Of the 11 respondents who only log in when they receive an e-mail alert, eight (73 
percent) rarely log into the system (that is, at the most a few times per month) and when 
they do log in, it is primarily because they received either a yellow or red alert. Only four 
respondents (out of the 11) actually utilize the alert module to explore the alerts received 
in their inbox. Furthermore, eight run syndromic or reportable diseases queries. Only six 
of the 11 respondents investigate alerts and five use ESSENCE to do so. A possible 
cause for the lack of ESSENCE use 1s because nine out of 10 of these respondents have 
not seen an actual outbreak using ESSENCE. Of the seven respondents who rarely log 
into the system and have disabled accounts, this could be attributed to ESSENCE not 


functioning they way they intended: by providing EED and SA. 


3 The chi-squared test was conducted on the cross-tabulation of question 4 with question 49 for rank 
categories E4—E6, O1—O3, O4—06, and civilian. The rank categories for E1-E3, E7—E9, and contractor 
were omitted because each category contained less than 10 respondents, and thus, resulted in multiple cells 
with 0 counts. All subsequent chi-squared tests involving rank were conducted similarly. 
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Q4: Do you log into ESSENCE only if you receive 


an e-mail alert? 
100° 0 


No 


o 
o ie 7 
o 


E! -E3 E4 - E6 E7 - ES O1-03 O4 - O06 Civilian Contractor 


Figure 3. | Percentage of respondents who log into ESSENCE only when they receive 
an e-mail alert by rank category. Overall, roughly 80 percent of the 
monitors do not log in just because they receive an e-mail alert. 


The next set of questions asked the respondents whether they ran reportable 
disease inquiries while logged into ESSENCE. As shown in Figure 4, greater than one 
out of two respondents (53 percent) indicated that they ran reportable disease queries 
routinely (1.e., at least a couple of times a week). As shown in Figure 5, there is a 
statistically significant difference (y7 = 26.876, p = 0.0298) by rank for those who 
conducted reportable diseased queries, with El1-E3s, O1—O3s, and civilians more likely 
to run reportable disease inquiries routinely. This is consistent with the fact that 
reportable diseases are primarily used in conjunction with NDRSi, and to confirm that 
their departments had submitted their medical reportable events on time, which are tasks 


not usually done by mid-grade officers in the U.S. Navy and Marine Corps. 
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9: On average, how often do you run a reportable diseases Query? 
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Figure 4. — Distribution of how often respondents ran a reportable disease query. 





Q9: On average, how often do you run a reportable discases Query? 
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Figure 5. | Frequency of how often respondents run reportable disease queries by rank. 
Across all ranks, 35 percent of the monitors routinely run reportable disease 
queries. 


In addition, there is a statistical difference between the percentage of monitors 


with an NDRSi account by user level (y° = 6.809, p = 0.0332)4 and by professional 


4 User levels (question 45) are single MTF, multiple MTF, and NEPMU level monitors. 
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background (y° = 15.275, p = 0.0016).5 In Figure 6, 100 percent of the NEPMU level 
users had an NDRSi account, while 26 percent of the single MTF level users did not. 
Furthermore, as shown in Figure 7, all of the physicians, and over 70 percent of the 
nurses, did not have an NDRSi account, while 75 percent or more of the other types of 
health professionals did have an account. Lastly, these same ESSENCE monitors did not 
verify the reportable diseases list they saw in ESSENCE with co-workers nor NDRSi 
because they did not have accounts and did not require them, nor will they in the future 


(Figure 8). 





Q10: Do you have a Navy Disease Reporting System, internet 
(NDRSi) account? 


100% 
80% 
60% 


BYes 


40% + 

















0% 
Single MTF level user Multiple MTF's level user NEPMU level user N-101 





Figure 6. By user level, percentage of respondents who have an NDRSi account. 


> Professional background categories (question 48) are preventive medicine technician, environmental 
health officer, preventive medicine officer, and nurse, corpsman, epidemiologist, and physician. The latter 
three categories were omitted from the chi-squared test because each category contained less than 10 
respondents, and thus, resulted in cells with zero counts. All subsequent chi-squared tests involving the 
professional background variable were conducted similarly. 
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010: Do you have a Navy Discase Reporting System, internet (NDRSi) account? 
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Figure 7. | Percent of monitors that have an NDRS1i account by professional 
background. An average of 80 percent of ESSENCE users have an NDRSi 
account. 


QI1: If confirmed, do you or do you with a co-worker verify that the 
reportable diseases you see in ESSENCE are also inputed into NDRSi? 
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Figure 8. | Percent of monitors that have an NDRSi account support facility. An 
average of 94 percent of ESSENCE users confirm that the reportable 
diseases that they see in ESSENCE are also put into NDRS1. 


As shown in Figure 9, 36 percent of the monitors routinely run syndromic queries, 
but 19 percent never use the module and 14 percent do not know what it is. Unlike 
reportable diseases queries, there is no difference in the frequency of syndromic queries 


by demographic groupings. However, the way the respondents used the data details while 
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running syndromic queries are statistically different (y° = 26.699, p = 0.0085),° namely 
for the data details parameter “age range” amongst the various rank categories (Figure 
10). One possible reason for this result, although not explainable from this survey, 1s that 
age range 1s not always important when narrowing down a clustering communicable 
disease 1n a given area. For instance, if an outbreak occurs on a ship, the narrowing down 
of the patients age range does not help the investigator track the outbreak, while the other 
parameters do; for instance, the clinics they visit (close to the docked ship) and the 
patient category (active duty personnel). Moreover, if the monitor deals mostly with 
active duty patients, age range would not matter, unlike some other monitor who 1s 


looking for an outbreak amongst children or the elderly. 


Q14: On average, how often do you run a syndromic Query? 
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Figure 9. _—_— Distribution of how often respondents ran a syndromic query. 


6 The chi-squared test was conducted on the cross-tabulation of question 16 with question 49. 
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Q16: When you run syndromic Query, how often are the following 
parameters important to your investigation? 
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Figure 10. Frequency of how often monitors use the age range parameter during an 
investigation. An average of 42 percent of the rank categories always use 
age range while investigation a particular syndrome. 


In terms of investigating alerts, 75 percent of the monitors investigate alerts 
(Figure 11), and approximately nine out of 10 (89 percent) of those use ESSENCE to 
conduct their investigations (Figure 12). There is a statistically significant difference (7 
= 8.875, p = 0.0029)’ by user level of those who use the system during their 
investigations. The majority of the multiple MTF level users tend to use ESSENCE, 
whereas 22 percent of the single MTF level users and 11 percent of the NEPMU level 
users do not use the system at all (Figure 13). One possible explanation for this 
phenomenon, which cannot be confirmed by the survey data, is that ESSENCE provides 
the user with a means to check multiple facilities at one time, even the entire Navy and 
Marine Corps (given the proper permissions), allowing the user to use one resource vice 
multiple external tools. ESSENCE gives the account holder the ability to observe alerts 
or potential outbreaks in a multitude of locations, thereby giving the monitor a starting 
point for an investigation. Single MTF users are more intimately familiar with the 
doctors and hospitals within their clinic or hospital and can speak with them more often, 
where a multiple MTF user does not have the ability to speak with physicians and nurses 


from all the hospitals as easily during their investigations. 


7 The chi-squared test was conducted on the cross-tabulation of question 20 with question 45. 
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Figure 11. Distribution of how whether respondents investigate alerts. 





Q20: Do you use ESSENCE to investigate alerts? 
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Figure 12. Distribution of whether respondent use ESSENCE to investigate alerts. 
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Q20: Do you use ESSENCE to investigate alerts? 
100% 











O0)% 











BYes 


40% 


20% 














Single MIF level user Multiple MTF's level user NEPMU level user N-94 


Figure 13. Percentage of respondents who use ESSENCE to conduct investigations 
about alerts. Overall, roughly 80 percent of the monitors do use ESSENCE 
to conduct their investigations of a potential outbreak alerted to them via the 

ESSENCE system. 


There is a statistical difference (y* = 28.246, p = 0.0051)8 by support facility for 
those who use FMP during an investigation. Illustrated in Figure 14, it is more likely that 
clinic and Naval Medical Center users will always use FMP, while NEPMU users are the 
least likely to use this data detail when investigating clustering within an alert. Over 60 


percent of the E7—E9’s and O4—O6’s always use FMP. 
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Figure 14. | Frequency of how often monitors use the data details, FMP, while 
investigating clustering within an alert. 


8 The chi-squared test was conducted on the cross-tabulation of question 23D with question 46. 
Support facilities are clinics, naval hospitals, naval medical centers, and NEPMUs. 
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Another data details functionality that was significantly different (y° = 17.737, p = 
0.0383)? by rank is how often monitors use the data details, MTF, while investigating 
alerts (Figure 15). One possible interpretation, which cannot be confirmed by the survey 
data, is that most of the rank categories understand how to make a site selection list and 
use that list when running a query to investigate alerts in their area. Rather than select a 
particular MTF during the query, they already specified their location by selecting the 
location through the “site selection” feature that limits their area of search during a 


syndromic query or within the alert list module. 


Q23: How often do you check the following data details to look for clustering 
within an alert? 
10° 
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Figure 15. Frequency of how often monitors use the data details MTF when looking for 
clustering within an alert by rank. 


There are also significant differences by user level (y° = 33.993, p = <0.0001),!° 
support facility (¥° = 38.181, p = <0.0001),!! and professional backgrounds 6a = 30.278, 
p = 0.0002)! with how often AHLTA is used during an investigation. Approximately 
three out of five (57 percent) of NEPMU level monitors rarely, if ever, use AHLTA as an 
external tool to aid them in their alert investigations (Figure 16). Furthermore, an 


average of two out of three clinic, naval hospital, and naval medical center monitors 


? The chi-squared test was conducted on the cross-tabulation of question 23H with question 49. 
10 The chi-squared test was conducted on the cross-tabulation of question 24A with question 45. 
11 The chi-squared test was conducted on the cross-tabulation of question 24A with question 46. 


12 The chi-squared test was conducted on the cross-tabulation of question 24A with question 48. 
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always use AHLTA to assist them in their investigations (Figure 17). One possible 
explanation for this result is that NEMPU level users and supporters do not have access 
to AHLTA and do not need it. NEMPU monitors are regional investigators who do not 
need to examine alerts with extreme detail, unlike the clinics, NH, and NMC users 
because they are examining their own area and part of their departmental duties, which is 
overseen by the regional staff members at the NEMPU level. Figure 18 illustrates that 85 
percent of the preventive medicine technicians always use ALHTA, while 50 percent of 
the physicians and environmental health officers do not. Part of preventive medicine 
technicians’ duties is to investigate alerts and use whatever means necessary to do so. 
AHLTA enables them to obtain the details of patient information and why they have been 
assigned a specific ICD-9 code. If the ICD-9 1s related to their investigation, then they 
can proceed towards determining if an actual outbreak is occurring. In comparison, 
corpsman may not have access to AHLTA and physicians may have already assigned a 
particular ICD-9 code to their patients or have spoken with fellow physicians, and 


therefore, do not need to verify this information in AHLTA. 


Q24: How often do you use the following external tools to further 


investigate clustering within an alert? 
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Figure 16. Frequency, by user level, of how often respondents use AHLTA as an 
external aid while conducting alert investigations. 
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Q24: How often do you use the following external tools to further 
investigate clustering within an alert? 
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Figure 17. Frequency, by support facility, of how often respondents use AHLTA as an 
external aid, while conducting alert investigations. 
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Figure 18. Frequency, by professional background, of how often respondents use 
AHLTA as an external aid, while conducting alert investigations. 


B. ASSESSING PERCEIVED VALUE 


I first begin by displaying the results of how often respondents applied for their 
current ESSENCE account, and how often they spoke with representatives while in the 
process of obtaining it. This is important due to respondent feedback through the open- 


ended comments sections of this survey. Distaste for constant password updates and 
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account requirement are prevalent throughout the comment sections and the distributions 
of survey results are not statistically different by user level, rank, professional 
background, and support facility. Due to an increase in operational tempo within the 
Department of Defense, military members are required to leave their billets often and 


must reacquire new accounts after 30 days if the program has not been logged into. 


In Figures 19-21, questions 35-37 illustrate the distribution of respondents who 
applied more than once for their current ESSENCE account, had to speak with an 
ESSENCE help desk representative, and the frequency of phone calls/e-mail needed to 
acquire the account. About 40 percent of users must apply more than once. On average 
one out of two (48 percent) had to speak with a help desk representative while waiting to 
access their current account. Finally, 45 percent contacted the help desk more than three 
times. In summary, users report what seem to be significant difficulties in acquiring an 
ESSENCE account; difficulties that surely do not engender feelings of good will toward, 


nor confidence in, the system. 


Q35: Did you have to apply more than once to obtain your 
current ESSENCE account? 
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Figure 19. Distribution of respondents who had to apply more than once to obtain their 
current ESSENCE account. 
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Q36: Did you have to speak with an ESSENCE helpdesk representative 
while wailing to aceess your current ESSENCE account? 
(Including e-mails and phone calls) 
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Figure 20. Distribution of respondents who had to speak with an ESSENCE helpdesk 
representative while waiting to access their current ESSENCE account. 





Q37: Estimate how many times you needed to contact the helpdesk (via e-mail 
or phone call) to acquire your current ESSENCE account. 
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Figure 21. Distribution of how many times respondents had to contact the helpdesk to 
acquire their current ESSENCE account. 


After examining the data to determine respondents’ perceived value of the system, 
I discovered that overall, the monitors believe that the ESSENCE system performs its 
situational awareness role well, but they find it lacking in its early event detection role. 
For example, Figure 22 shows that three out of four respondents have not discovered an 
actual outbreak using ESSENCE. There is a statistically significant difference (y* = 
12.363, p = 0.0062)!3 by rank of monitors who discovered an actual outbreak using the 


13 The chi-squared test was conducted on the cross-tabulation of question 29 with question 48. 
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biosurveillance system. Over 60 percent of the rank categories except O1—O3s have not 
used ESSENCE for early event detection (Figure 23). The interpretation is that the 
majority of the respondents believe they did not see an actual outbreak occur although 
ESSENCE indicated that one existed. Further proof that respondents tend to see more 
false positives exists when a cross-tabulation of question 29 was conducted with question 
33H, which asks respondents if they agree or disagree with the statement: “I find that the 
alerts in ESSENCE are often false-positives.”” Nine out of 10 monitors (Figure 24) who 
said they have not discovered an actual outbreak said they agree or are neutral to this 
statement. A few respondents who have not discovered an actual outbreak using 
ESSENCE said they would like “training and insight into how syndromic alerts are 
derived” and “how to detect false positives.” Another monitor stated that “understanding 
how the alert functions would be better [to know]. Right now, I feel that the alerts are set 
to be too sensitive, as we get quite a few of them here. Investigating all of them would 
take far too much time. When we do investigate them, we never come to a conclusion on 
the cause (if any) for the increase in patients for that particular syndrome.” Finally, one 
user said that they would like to “ have [a] person [who 1s] doing the [application] 
screening [to] ask for all the additional info needed at once instead of responding (a week 


later) after one item has been submitted then asking for something else.” 


Q29: Have you discovered an actual outbreak using ESSENCE? 
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Figure 22. Distribution of respondents who say they have discovered an actual 
outbreak using ESSENCE. 
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Q29: Have you discovered an actual outbreak using ESSENCE? 
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Figure 23. Percentage of monitors, by, rank, who have discovered an actual outbreak 


using ESSENCE. Roughly 75 percent of the respondents said they have not 
seen an actual outbreak in ESSENCE. 
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Figure 24. Percentage of monitors who have not discovered an actual outbreak using 
ESSENCE but who also either are neutral or agree with the statement that 
they find that ESSENCE alerts are usually false-positives. 


Of the 33 respondents who have observed an actual outbreak in ESSENCE, over 
20 confirmed actual cases of ILI, GI, and respiratory outbreak types as well, seven have 


seen heat, febrile, neurological, pertussis, and exposure to chemical/toxins cases. 
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However, 75 percent of the respondents observe miscoded ICD-9 codes in 
ESSENCE (Figure 25). Miscoding causes alerts to increase because providers, 
immunizations representatives, MTF coding representatives, or nurses have input the data 
incorrectly into the system and currently these mistakes cannot be removed or updated by 
monitors who have verified the miscodings using AHLTA or CHCS. Of the 100 
respondents who witness miscodings, 73 percent agree that it would be easier to use 
ESSENCE if a patient’s final diagnosis was viewed differently in the system, perhaps if 
bolded. In addition, 95 percent agree that it would be easier to use ESSENCE if a 
patient’s miscoded ICD-9 codes were updated and correctly input into the system. Again, 


miscodings are a potentially significant cause of the false-positives within the system. 


Q31: Have you observed a miscoded ICD-9 code in ESSENCE? 
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Figure 25. Distribution of respondents who have observed a miscoded ICD-9 code in 
ESSENCE. 


Question 33 asks respondents whether they agree or disagree with eight 
statements regarding the ESSENCE program, presented in Figure 26. An average of 92 
percent of the respondents said they value the system; that is, they tend to strongly agree, 
agree, or are neutral towards all eight statements. However, it is important to understand 
why the other 8 percent do not value the system and what can be done in the future to 
remedy their perceptions, issues, or problems in order to make the system valuable to all 


future users. 
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Distribution of responses regarding respondents perceived value of the 


Figure 26. 


ESSENCE system. 
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Roughly 85 percent of the monitors feel comfortable using the ESSENCE 
program. On the other hand, over half of the 15 percent of respondents who do not feel 
comfortable with the system believe that ESSENCE does not provide them with SA 
about communicable diseases in their area and they do not think that bolding a patient’s 
final diagnosis would make ESSENCE easier for them to use. Of these 20 individuals, 
70 percent log in rarely (that 1s, monthly to never), 50 percent are multiple MTF level 
users, roughly 40 percent are preventive medicine technicians ranging in rank between 


E4—E9 and are split between having active and disabled ESSENCE accounts. 


Figure 27 indicates there is a statistical difference (y° = 16.107, p = 0.0409)!4 by 
user level of those who think ESSENCE provides their department with situational 
awareness about communicable diseases of interest in their area. Twenty-seven percent 
of the NEPMU level users disagree with this statement, while 90 percent of the MTF 
level users agree that ESSENCE provides them with a suitable amount of situational 
awareness about outbreaks in their area. The chi-squared test conducted on the cross- 
tabulation between this statement and rank, a statistically significant result (y° = 25.873, p 
= 0.0112) indicates that 24 percent of the O1—O3s and 10 percent of the O4—O6s do not 
feel that ESSENCE provides their department with situational awareness either, as 


depicted in Figure 28. 


Q33: Llow much do you agree or disagree with the following statements 
regarding the ESSENCE program 
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Figure 27. Percentage of monitors by user level who agreed, was neutral, or disagreed 
with the statement that ESSENCE provides their department with situational 
awareness about communicable diseases of interest in their area. 


14 The chi-squared test was conducted on the cross-tabulation of question 33B with question 45. 
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Q33: How much do you agree or disagree with the following statements regarding the 
ESSENCE program 
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Figure 28. Percentage of monitors by rank who agreed, was neutral, or disagreed with 
the statement that ESSENCE provides their department with situational 
awareness about communicable diseases of interest in their area. 


Similarly, monitoring trends of diseases in a certain location over time allows a 
monitor to track and perhaps understand the local trends in the diseases. There is a 
statistically significant difference by support facility (y° = 29.292, p = 0.0036)!5 and rank 
(y° = 26.887, p = 0.0081)! of those who do not believe that ESSENCE allows them to 
monitor trends of diseases over time, which provides the situational awareness that is a 
key role for ESSENCE. Similar to the results in Figure 27, Figure 29 shows that 20 
percent of the NEPMU support facilities do not think ESSENCE allows them to monitor 
the trends of diseases in their area over time. Furthermore, 17 percent of the O4—O6s and 
small number of Ol—O3s and civilians believe that this ability 1s lacking within the 
program (Figure 30). On the other hand, seven out of 10 users think ESSENCE is useful 


for recognizing trends over time, and believe SA is being provided to them. 


13 The chi-squared test was conducted on the cross-tabulation of question 33C with question 46. 


16 The chi-squared test was conducted on the cross-tabulation of question 33C with question 49. 
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Figure 29. Percentage of monitors by support facility who agreed, were neutral, or 
disagreed with the statement that ESSENCE allows them to monitor the 
trends of diseases in their area over time. 
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Figure 30. Percentage of monitors by rank who agreed, was neutral, or disagreed with 
the statement that ESSENCE allows them to monitor the trends of diseases 
in their area over time. 
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In addition, there was a statistical difference (y° = 16.569, p = 0.0349)!7 by user 
level or those who believe that ESSENCE allows them to follow particular reportable 
diseases in their area. Once more, NEMPU level users disagree that they are able to 


follow reportable diseases in their area (Figure 31). 
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Figure 31. Percentage of monitors by user level who believe that ESSENCE allows 
them to follow particular reportable diseases in their area. 


Through exhaustive research of the data, two questions in this question that 
identify outbreaks and false-positives are the most important. Inherently, ESSENCE is 
an outbreak detection tool, mostly for EED, and without collaborating programs, false- 
positives are time consuming for monitors to investigate. Of the 11 respondents who do 
not think that the time-series module is useful for identifying outbreaks, each one did not 
see an actual outbreak in ESSENCE. Roughly 70 percent of these monitors are routine 
users of the program, 54 percent routinely use the reportable diseases query module and 
27 percent routinely use the syndromic query module as well. Six respondents use 
ESSENCE to investigate alerts and about 50 percent perceive the system to be ineffective 
and not valuable. Some reported that they use “AHLTA, CHCS, and labs” more often 
than ESSENCE, and their biggest headache comes from gaining “access” to the program 


17 The chi-squared test was conducted on the cross-tabulation of question 33D with question 45. 
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in the first place. They would prefer that “ESSENCE communicate with programs, such 
as AHLTA and CHCS,” so they would not have to investigate false-positives and one 
monitor did not have “access to Personal Health Information (PHI) information,” 
although access was requested multiple times. Finally, users reported that they could use 
ESSENCE for “situational awareness, but not for outbreak detection” and it was “not 


useful during the H1N1 outbreak.” 


Lastly, 62 respondents said they find that alerts in ESSENCE have a propensity to 
be false-positives. Sixty-six percent are routine login users, 62 percent run reportable 
diseases queries and 50 percent run syndromic queries. Approximately 77 percent use 
ESSENCE to investigate alerts, but only 33 percent has detected an actual outbreak using 
the system. A few have requested “more training,” primarily “hands on training that 
provides practical use in one’s department,” such as “describing how [the] alert function 
works,” outbreak scenarios and times-series module analysis. Some respondents recalled 
that ESSENCE “times out too quickly” and constant password updating makes the 
account holder spend more time speaking with ESSENCE help desk representatives, 
rather than logging in and checking for alerts. Finally, some feel that the “daily 
requirement to check ESSENCE seems to be a bit much.” One respondent commented, 

I find that many times the alerts are false alerts and that the “syndrome” 

parameters are so wide that sometimes the alert doesn't seem credible. 

(1.e., a rash in two infants will trigger an alert and so will ptosis of the 

eyelid diagnosed in the optometry clinic.) I will check ESSENCE and 

then check the AHLTA encounter and contact the providers or clinics to 

[obtain] “ground truth” the data to see if what I see in ESSENCE 1s truly 

what is happening in the clinics. I do use ESSENCE to see how we are 

doing with regard to trends in coding for certain things like ILI, Resp, and 

GI syndromes. I share this with providers as appropriate. I also check the 

reportable disease section and use this as a “safety net” to catch any 

reportable diseases that weren't consulted to Prev Med or didn't show up 

as a positive lab in our box in the Laboratory. We follow up on each 


reportable disease that shows up in ESSENCE. At times we will find 
STDs and other reportable diseases this way. 


In conclusion, at least nine out of 10 respondents value ESSENCE, but the 8 
percent who do not value the system have important recommendations and reasons why 


they do not favor the system. Most of them center on better and specific training, 
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constant password renewal, false-positives that decrease actual outbreak detection, 
miscodings, and compatibility with other external systems, such as AHLTA and CHCS, 


to view patient information while logged in. 


C. ASSESSING USER TRAINING 


A series of questions were developed to examine how ESSENCE monitors were 
trained to use the system and what they would like to see in the future. There are no 
statistically significant differences in how the respondents where trained by user level, 
support facility, professional background. Seventy percent of the respondents were self- 
taught, followed by 34 percent who were trained by their processors or downloaded the 


ESSENCE training modules and tutorials, and 21 percent were taught by their bosses 
(Figure 32),!8 


Q38: Who did you receive your ESSENCE training from? 
Check all that apply. 
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Figure 32. Distribution of where respondents received their ESSENCE training. 


With no statistical difference (by rank, user level, professional background, and 
support facility), the majority of the monitors would like to see future training provided 
to them at their desks, in public health conferences, in hospital/clinic meetings, “C” 


school, NEPMU, and CME training. However, 30 percent do not want to have meetings 


18 Percentages add up to more than 100 percent because respondents could select more than one 
response. 
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at the hospitals/clinic meetings, indicating (not verifiable through survey data) that 
meetings during working hours are not conducive to ESSENCE training and should be 
conducted elsewhere. Fewer than 20 percent of users do not want to see training at their 
desks or in health conferences, which is evidence enough that these would be better 


venues for ESSENCE training. 


Ninety-seven percent of the respondents reported that they would prefer hands-on 
training. There were no significant differences by rank, professional background, user 
level, or support facility. Furthermore, at least nine out of 10 monitors would like to 
receive initial ESSENCE user training, including training for site selection set up, how to 
evaluate alerts, query module applications, and outbreak scenarios. A few individuals 
also wrote that they would like to learn how to “detect probable false positives,” “use the 


matrix function,” and “time-series explanation and 14-day incidence explanations.” 


Q39: Where would you like to see training performed for current 
and future users of the ESSENCE system? 
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Figure 33. Distribution of where respondents would like to receive their ESSENCE 
training in the future. 
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Figure 34. 


100% 


80% 


60% 


40% 


20% 


0% 





Figure 35. 


Q40: How would you like to see training provided for current 
and future users of the ESSENCE system? 
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Distribution of how respondents would like to receive their ESSENCE 
training in the future. 
Q41: What type of training would you like for current and 
future users of the ESSENCE system? 
No 
MYes 
Initial training to Site selection set How to evaluate Query module Outbreak scenarios 
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Distribution of what respondents would like to see in ESSENCE training in 


the future. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY AND CONCLUSIONS 


The Navy and Marine Corps use the ESSENCE system for early detection of 
diseases and other public health threats to the force and for situational awareness on the 
location and spread of such diseases. DoD Directive 6490.2E dated August 24, 2009 
requires that “the DoD components shall conduct comprehensive, continuous, and 
consistent military health surveillance to implement early intervention and control 
strategies, using joint technologies, practices, and procedures in a manner consistent 
across the military services. Relevant, timely, actionable, comprehensive health 
surveillance information shall be collected and maintained to support the Armed Force.” 
Following this guidance in 2009, the U.S. Navy and the Navy Bureau of Medicine and 
Surgery (BUMED) mandated that the “monitoring of ESSENCE alerts applies to all 
Navy medical treatment facilities (MTFs). Health surveillance shall be conducted to 
enable early intervention and control strategies. Monitoring for and timely reporting of 
significant health events that may adversely affect mission accomplishment and shape the 


commander’s decision making 1s vital to military medicine functions. 


Navy ESSENCE monitors are health professionals, including preventive medicine 
technicians, doctors, nurses, public health professionals, and epidemiologists. The 
agencies that monitor ESSENCE are the preventive medicine and public health 
departments of the Navy, which are spread throughout the world. They are comprised of 
naval clinics, naval hospitals, and naval medical centers. The Navy Environment 
Preventive Medicine Units (NEPMU), which are located in San Diego, California, Pearl 
Harbor, Hawau, and Norfolk, Virginia, also monitor disease activity using ESSENCE. 
Finally, the Navy and Marine Corps Public Health Center (NMCPHC) oversees patient 
syndromic surveillance throughout the Navy as mandated by BUMEDINST 6220.12B. 


This survey, conducted on behalf of the NMCPHC, was sent to 225 Navy and 
Marine Corps users with either an active or a disabled ESSENCE account. The survey 


was in the field for eight weeks. A total of 143 monitors (64 percent) responded to the 
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survey and seven (3 percent) opted out of taking it. Survey findings include that, overall, 
92 percent of ESSENCE account holders, past and present, favor using the system, find it 
valuable, and believe the training they received was adequate. However, users raised 
four main issues: 1) it takes an excessive amount of time to obtain an account, 2) 
passwords are required to be changed too often, 3) there are too many miscodings leading 


to excessive false positive signals, and 4) training and training tools are insufficient. 


First, acquiring an account and maintaining it 1s a recurrent problem among the 
users. Also, once the account has been given, users must change their passwords every 


other month. One respondent took the time to follow-up in an e-mail and wrote that: 


the stringency of the 15 digit complex password that has to be re-set every 
60 days. I find myself spending an incredible amount of time “trying out” 
new passwords...only to discover it may have failed for lack of a second 
numeral, etc....and then have to do it all over again in 60 days. I do have a 
few “tricks” that help, but still haven't found a foolproof way to satisfy the 
requirement in a time-effective manner. I understand the need for 
security--- perhaps re-setting the pw every 90 days might be a reasonable 
compromise between maintaining a high-level of security and trying to 
increase user friendliness. 


Another respondent commented in the survey: 


Don't ask for me to change my password as often as requested. I have 
been on leave or TAD twice when my account has been inactivated. 
Although I should have completed the task before I left, I don't change my 
bank info as much as I have to change ESSENCE. I understand the 
privacy issue, but most medical folks not involved in Preventive Medicine 
wouldn't even know where to access ESSENCE b/c they don't know what 
itis. I think it's a secure site, but I have had to reset my password several 
times. 


Understanding that safeguarding of personally identifiable information (PII) is 
very important, but a single standard should be in place for all programs with PII, 
meaning that password change requirements should be the same length as AHLTA or 
CHCS. If passwords need to be revised often, then is it important to notify the users of 


this and the reasons. A well-informed account holder is a happier one. 


Second, numerous comments declared that miscodings are too prevalent and 
respondents believe they contribute to a high false positive rate in ESSENCE. One 
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monitor states, “there are [too] many false positives, and many of the abbreviations 
remain a mystery.” The user suggests, “a key or legend would help,” and if one exists, to 
“make it more apparent to access. The idea of the program is marvelous --- so if it was a 
little easier to use I think more of us would use it as it was intended.” Another well- 
informed user, but one that believes false positives are numerous, said that: 

The daily requirement to check ESSENCE seems to be a bit much. I find 

that many times the alerts are false alerts and that the “syndrome” 

parameters are so wide that sometimes the alert doesn't seem credible. 

(1.e., a rash in 2 infants will trigger an alert and so will ptosis of the eyelid 

diagnosed in the optometry clinic.) I will check ESSENCE and then 

check the AHLTA encounter and contact the providers or clinics to 

‘ground truth’ the data to see if what I see in ESSENCE is truly what is 

happening in the clinics. I do use ESSENCE to see how we are doing with 

regard to trends in coding for certain things like ILI, Resp, and GI 

syndromes. I share this with providers as appropriate. I also check the 

reportable disease section and use this as a ‘safety net' to catch any 

reportable diseases that weren't consulted to Prev Med or didn't show up 

as a positive lab in our box in the Laboratory. We follow up on each 

reportable disease that shows up in ESSENCE. At times, we will find 

STDs and other reportable diseases this way. (Sometimes we find that they 

are simply rule-outs or coding errors, 1.e., 'meningitis' was really a 'part 2 

sea duty screen.’ Which keeps things interesting [sic].) 

I am not suggesting that false positives are common, or that the system's 
sensitivity is too high, but ESSENCE account holders believe they are. They are 
requesting training, in particular, how to detect outbreaks and how to search through the 


alerts and false-positives for an actual alert. 


Third, training will likely mitigate at least some of the problems that have been 
described. Many of the respondents suggested that “hands on training in practical terms 
for your particular command/MTF” 1s preferred. Although ESSENCE users have had 
initial training on how to log in and prepare their site selections to begin their 
investigations and to use the alert module and run syndromic and reportable diseases 
queries, most do not understand what the time-series tool contributes to and how to 
properly search through the alerts to detect an outbreak. Practical lessons should be 


developed to help educate current and future users. 


a/ 


B. RECOMMENDATIONS 


To mitigate the password issues one recommendation that can be made is to 
modify ESSENCE to allow CAC enabled access. This would eliminate the need for 
users to have to reset passwords on a very frequent basis, which will ultimately increase 
the value and desire to use the system. This approach allows the frequently deployed 
service member the ability to gain access once they arrive back into their positions to log 
into ESSENCE without reacquiring an account and wasting up to a month before they 


can begin using ESSENCE. 


To help address the issue of miscodings, perhaps modify ESSENCE to allow 
corrections to be made by the account holder who has access to CHCS and AHLTA, 
providers who verify the update, and by the laboratory technicians. If the users have the 
ability to change the miscodings to the correct codes, then perhaps, this may help bring 
down the alerts rate while improving ESSENCE sensitivity to detect actual outbreaks. 
Another possible fix for miscodings is to allow the individual who originally input the 
data incorrectly to change and delete the old entry. Or as stated in the survey, which 96 
percent of the respondents agreed with, is to bold the finalized ICD-9 code and syndrome 
that will allow the ESSENCE monitors to focus their investigations on actual PIN 


information and alerts. 


Lastly, in terms of training, practical applications either via Web-based video 
training or hands on training in the workplace will substantially increase user 
understanding in the system and, hopefully, will decrease their beliefs that false positives 
are the only thing that ESSENCE “detects.” Users generally believe ESSENCE is a 
valuable system, and with the proper tools, account holders will be able to detect 
communicable diseases earlier in their area and provide the necessary situational 
awareness about these same diseases to enhance the security of the United States and 


throughout the world with healthy members of the military and their families. 


C. FUTURE RESEARCH 


This survey was only administered to Navy and Marine Corps active and disabled 
account holders, but I recommend that future research focus on Air Force and Army 
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account holders to see if the problems areas are similar and if not, are they better or 
worse. If they are better, then perhaps joint implementation of ESSENCE could be 
taught to the different military services to continue to improve the system and allow users 
to value it more. If all members of the Armed Forces have the same main problem areas, 


then a fix in the system and thorough training can be provided to everyone. 
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Ql 


APPENDIX A. USER SURVEY 


Official ESSENCE User Survey (BUMED 6220-3) 


ABOUT THIS SURVEY 


Introduction. You are invited to participate ina survey lo help thea Navy ane Marina Corps Public Health Center (NMOPHC) better understand 
how tha Electronic Surveillance System for Early Notification of Community-based Epidemics (ESSENCE) is baing implemented and used in 
the Navy and Marine Caras. Tha purpose of tna survey is to provide NMOPHC with information about: 


« Whethe and how users hava been traned to use ESSENCE: 
# How ESSENCE is currently baing amployed by the Navy and Marina Corps: and, 
e How effective users think ESSENCE is for outbreak detection and situational awareness. 


Procedures. This web-based survey is being conducted by the Naval Postgraduate School (NPS) on behalf of NMCPHC. It should take no more 


than 20-25 minuies to CoONVelE. 


Risks. The potential risks of participating in this study are inadvertent disclesure of individual survey responses. To mitigate this risk, the project 
has impleameniad extensive dala safeguarding procedures: a capy of Ine Datla Safequarding Plan is available from (he survey praject leader, 
LT Randi Korman. Yet, even with such procedures in place, there always ramains some risk, however small, of a data breach. 


Benefits. This study is designed to provide NMCPHEC with recamnendatiors to improve ESSENSE performance, training, and usability of the 
system. This is your opportunity to provide NMCOPHC with your fravk and honest faedback so decisions can ba mada to most benefit currant and 
future ESSENCE users. 





Compensation. No tangible compensation will be given. /f pou would like fo receve a copy of the report, send an email to mmborman@aps. edu 
With the werd UMCLUDE' in fhe subject Mine and you will be added to the ofsiribution list. The results wil be available in the March 2011 
timeframe fram WMCPHC via Ms. Asha Riegededios, (904) 542-4535, asha riagodadiss2 2 med. navy.mil. 


Confidentiality & Privacy Act. Any personally-identifying information that is obtained during ths study will be kept confident al ta the full 
extent permitted by law and your participation in the survey will not be disclosed outside of the NPS team. Your suey responses may be 
merged with demographic data provided by tha NMCPHC for purposes of analysis. All efforts willbe made by NPS to keap your personally 
identifying information cenfidential, but total confidentiality canrot be guaranteed. lt is possible that NPS is raguited ta divulge information 
obtained in the course of this reasearch to the subject's chain of command o° other legal body. Lpon completion of the survey. the team will de- 
idantfy the survey results to ensure complete anonymity and the da-identifiad data will be provided to NMCPHC. 


Voluntary Nature of the Study. Participation in this study is stricth voluntary. and if agreement to participation is given, it can be withdrawn at 
any time without prejudice. 


Poinis of Contact. Plaase contact he survey project leader and co-investigater, LT Aandi Karran, 360-929-9845 imbkorman@nps.adu, with 





any questions about this survey. Tha principle investigator is Or. Fon Fricke, 831-656-3048, rdffickea@inps.edu. Questions about your rights as a 
research subject or any other concerns may be addressed to the Navy Postgraduate School IRB Chair, GAPT John Schmidt, USN, 831-€56- 


S876, jkschmid@inps.edu 


Please click onthe "YES" button to indicate that you voluntarily agree to take the 
survey. 


C YES C NO 


In this section we are interested in how you are currently using the ESSENCE system. 
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Q2 


Q5 


Q6 


Q7 


On average, how often to do you log into ESSENCE? 


Daily 

Weekly (a couple times per week) 

Monthly (a couple times per moth) 

Greater than monthly (a couple times per year) 


Neawier 


Do you receive e-mail alerts from ESSENCE? 


C Yes 


€ | didn'tknow | could receive e-mail alerts 


Do you log Into ESSENCE only If you receive an e-mail alert? 


C Yes 
Ha 
Which e-mail alert(s) do you receive? 
Yellow alerts 
Aad alarta 


Beth alerts 


I don't anew 


Please explain why you do not receive e-mail alerts. 


In this section we are interested in learning if you use the Alert List. 


Do you monitor the Alert List module? 


C Yes 


| dan't kantw what this = 





Cin = 
r ou 
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Q8 


Q9 


Q10 
Q1l 


(12 


Please explain why you do not use the Alert List to monitor alerts. 


El 











gl 
In this section we are inierested in how you run a reportable disease Query. 
On average, how often do you run a reportable diseases Query? 

Daily 

Weekly (a couple times per week) 

Monthly (a couple times per month) 

More than monthly (a couple times par year) 

Newer 


| don't Know what this is 


Do you have a Navy Disease Reporting System, internet (NDRSi) account? 


If confirmed, do you or do you with a co-worker verify that the reportable diseases you 
see in ESSENCE are also inputed into NDRSi? 


How much do you agree or disagree with the following statement? 
| find that most often the reportable diseases | see in ESSENCE are rule-outs. 


= 


{ Strongly Agree 
“ Acree 
Neutral 


Disagree 


Strongly Disagree 
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Please explain why you do not use the reportable diseases Query module. 


Q13 : 


In this section we are interested in how you run asyndromic Query module. 


On average, how often do you run a syndromic Query? 


Q14 — 


Weekly ‘a couple times per week} 

Monthly (a couple tires per month) 

More than monthly (a couple times per year] 
Never 


| dont Know what this is 


When you run a syndromic Query, how often do you use the following data sources? 


Q l 5 Always Frequently Sometimes Rarely Never 


SADA C Cc 8 2 C 
o c 


Phannacy 


c c c 
Laboratory c C o o c~ 
cc cc oF 


rl rt 


Fadiology ( ( 


When you run syndromic Query, how often are the following parameters important to 


Q l 6 your investigation? 


Always Frequently Sometimes Rarely 
Age Range c C) 
Clini: Type 
Start Date 
End Date 
Patiant Salegory 


Patient Military Branch 


For a syndromic Query, do you do the following during your investigation? 
Q l 7 Yos 

lnarrew dawn (or epocity) patient parameter dapending co 

an the ayndramesfdiaeaas | am inveatigat ng 

llocd al syndramis patterns over ime Cc 


linvestigata cartain alerts clsplayad on ihe data table 
by clicking an the Data Details for a given day 
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Q18 


Q19 


Q20 


Q21 


Q22 


Please explain why you do not use the syndromic Query module. 


— 





a 


In this seclion we are interested in learning aboul the process of how you investigale an alert by using 
either the Alert List or the Query module. 


Do you investigate alerts? 


When you begin an investigation using ESSENCE, which of the following site selections 


do you choose? 


Your local MTF 

Srly Mavy and Marine Cerss MITF's in 
Yor regian 

All services’ MTF's 17 your regan 

Across all Navy and Marine Corps MTFs in 
YOLF country 


Across all services’ MTFs in your country 


Once you notice a yellow or red alert for a particular syndrome, do you look up data 
details within ESSENCE to verify the need for additional investigation. 


{ Yas 


{ Wo 
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How often do you check the following data details to look for clustering within an alert? 


Q23 Always Frequently Sometimes Rarely Never 


PIN C C L 
Encounter Dete 
Age 

FF 

ICD-9 Descriptions 
Clinic Type 
Provider 

MTF 


PatGal (Patient Category) 


ERO Oo RB Oo Reo 
ORB ORB OLB O reo 
ORD ORB Oo CEO Reo 


Look at the map view for a 


spatial representation 
Other plaase spacity) 
How often do you use the following external tools to further investigate clustering within 


Q24 angie 
Always Frequently Sometimes Rarely 


AHLTA Cc cc f { 
CHES 


Speak with providers 


o 2 
© e 
~ o 


teciniciars 
Speak with immunizations 
teciniciars 


io 
Cc 
Speak with laboratory c 
ti 


Other (plaase spect) 
Po 
Do you maintain a log or take notes (in Word or Excel) of an investigated alert for future 


Q2 5 reference? 


C Always 
{ Depends cn the syndromedisease investigated and/or the severity of the outbrazk investigated 


( Never 


Q2 6 After researching an alert, do you bricf your command? 
{ Alwaye 
f Depends cn the syndromedisease investigated andiar the severity of the outbrock investigated 


f Newer 
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Q27 


Q28 


Q29 


Q30 


Please explain why you cdo not investigate alerts. 


r 


lf you do not use ESSENCE to investigate an alert, what alternate steps do you take to 
conduct your investigation? 


r 


ein chis sectioid we ae interested in understanding outbreak detection while using ESSENCE. 


Have you discovered an actual outbreak using ESSENCE? 


lf you have answered yes to the above question, check all the outbreak types you have 
discovered using ESSENCE. 


rT ui 
Heat 
Gl 
Respiratory 
Fetrile 
Neurological 
Rash 
Hane 


Other (please spacily) 


a 


In this section we are interested in how you handle miscodings. 
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Have you observed a miscoded ICD-9 code in ESSENCE? 
Q31 C Yes 


{ Wo 


What steps do you take to [fix miscodings? 


Q32 Check all that apply. 


. I|speak with the provider 

[ [speak with the immunizations represanative 

| Ispeak with the MFT coding representative 

[ speak wilh the preventative medicine representative al the clinic/hospita 
. Idone speak wih anyone 


Other (please spacily) 


In this section we are interested in your opinions of the ESSENCE system. 


How much do you agree or disagree with the following statements regarding the 


Q33 ESSENCE program? 


Strongly Stranrcghy 
Agraa Neutral Disagree 
Agen Nisagrae 


| feel comfortable using the ESSENCE program c o 


ESSENCE provides my department vith situational awareness about communicable c c 
diseasas of irterest in my area 


ESSENCE allows me te monitor tve trands of ciseases ir my araa ove: time cr" t 


ESSENCE allows me to folow particular reporiabls diseases in my area C {— o C 


ESSENCE would be easier to use if a patient's final diagnosis was viewed dilfarently 
in tie system, perhaps if bolded 

ESSENCE woule be easier to use if a patient's miscoded ICD-9 code were updated 
and correctly inputted inte the system 


The timeterice nedule in ESSENCE is useful te identify ouibreake 


| find thatthe alerts in ESSENCE are offen falea-positives 


In this section we are interested in learning about your experience in obtaining your current ESSENCE 
account 
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Q34 How long did it take you to acquire your current ESSENCE account? 
C 1+15 days 
16+ 30 days 


41 - 45 days 


0 
0 
{ 46-60 davs 
O 


More than 60 days 


{| den't remember 


Did you have to apply more than once to obtain your current ESSENCE account? 
Q35 C Yes 


{ Wo 


Did you have to speak with an ESSENCE helpdesk representative while waiting to 
Q36 access your current ESSENCE account? (Including e-mails and phone calis) 


{Yas 
f Wo 


( | don't remember 


Estimate how many times you needed to contact the helpdesk (via e-mail or phone call) 
Q37 to acquire your current ESSENCE account. 


1 time 

2 times 

4 times 

4 times 

Bor mare ines 
{ | don't remember 


In this section we are interested in learning about what training you have had on the ESSENCE system 


and what you would like to see in the future. 
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Who did you receive your ESSENCE training from? 


Q3 8 Check all that apply. 


[ My predecessor 
My boss 


lam eelf-Laughit 


latterded a conference where | waa taught how to work che ayatem 


- 
Ez 
[ | downloaded the ESSINCE training modules and tutor als 
J 
- 


| wae taught by che NESMU 


Ciher (please epocity) 


[ 


Where would you like to see training performed for current and future users of the 


Q3 Q ESSENCE system? 


Al my own dask 

In public health cantarances 
In hespitalfclinis meetings 
In" schol 


Mikes [plasse saecify) 


L 


Q40 How would you like to see training provided for current and future users of the 
ESSENCE system? 








Audio and video based 
Powerpe nl preasearcatiana 
Hands-on training 


Cine (please sgecily) 


| | 





70 


Q41 


Q42 


Q43 


Q44 


What type of training would you like for current and future users of the ESSENCE 
system 


Initial taining ta uae ESSENCE 

Site selection sel up 

How tc evaluate alerts 

Quary modules applications 

Qulbrek scenarias 

Other please specify) 
Po 


Do you have any suggestions that will help make the ESSENCE system easier to use? 


Do you have any suggestions that will help make the ESSENCE system more effective? 


Do you have any other comments that you would like to make regarding the ESSENCE 
system? 


In this section we are interested in learning about you as a monitor. 
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Select the current user level that best describes you. 


Q45 C Single MTF level user 


{ Multiple MTF's level user 


( NEPMU level user 


Ciher (please epecity) 


| | 


Select the facility that you currently support. 


Q46 ro 


( Naval Hospital 








( Naval Medical Concer 


 NEPMU 


Ciher (please specify) 


SS Ss 
Q 47 Check all the service(s) you currently support. 


| U.S. Air Foree 
[ U.S. Army 


. U.S. Marine Corps 


[|] us. Have 


Q 4 3 Select the professional background that best describes you. 
Cerpsman 
Preventative Medicine Technician 
Epidemiclagist 
Ervironrrental Health Officer 
Preventative Medicine Officars 
Nurse 
Pry siciar 


Olher (please specity) 


esl 





p 


Which category do you fall into? 


Q49 7 


- EG 


-O9 
OG 
Civilian 


Coantracter 


lf you are acivilian of a conlractor, please enter your grade level or aquivalant (GS-7, WG-8, WL-11, ete) 


| | 


Thank you for completing this survey on behalf of the Naval Postgraduate Scheol and the Navy and Marine Corps 
Public Health Center. 
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APPENDIX B. IRB DOCUMENTS 


Naval Postgraduate School 
[nstitulional Review Boord (1B) 





From: President, Naval Pomgraduate Schoo! 
Via: Chairman, Institutional Review Board OCT 2 2 Bl | 
To: Dr. Ronald Fricker, Jr, Operations Research Deparumem 

1.T Randi Korman, (ISN 


SUBJ: ASSESSING THE ESSENCE BIOSURVEILLANCE SYSTEM AS USED BY 
THE NAYY AND MARINE CORPS: USER TRAINING, SYSTEM 
EMPLOYMENT AND PERCEIVED VALUE 


Encl: (1) Approved IRB Protocol 


|. The NPS IRB is pleased to inform you that the NPS President has approwed your 
project (NPS IRB NPS.2011.0001-[R-EP?-A). The approved IPRLB Protocol is found in 
enclosure (1), Completion of the CIT] Research Ethics Training has been confirmed. 


2. This approval expires on 3] March 11. If additional time is required to complete the 
research, a continuing review report must be approved by the [RB and NPS President 
prior lo the expiration of approval. Al expiration all research (subject recruitment, data 
collection, anulysis of data contaming Pll) must cease. 


3. You are required to report to the TRG any unanticipated problems of serious adverse 
events to the NPS IRB within 24 hours of the oocurrence. 


4. Any proposed changes in IRB approved research must be reviewed and approved by 
the NPS IRB and NPS President prior to implementation except where necessary lo 
eliminate apparent inmumediate hazards to research participants and subjects. 


5. Ag the Principal Investigator it is your responsibility to ensure that the research and 
the aalions of wll project pease! involved in conducting this study will conform with 
the IRB approved protocol and IRB requirements/policics. 


6 Afler the experiment ts completed the Poncipal Investigator wall submit to the Human 
Subpects Protection Office, all signed informed consent documents, unanticipated 
problem reports, adverse event reports and a End of Experiment Report. The Homan 
Subjects Research CNfice wall secure these documents for 10 years and then forward to 
mearest FRCL. 




















. a fof ia ¢ 
ie = 5, ne eS are = i ; : 
FAP T John Schimedt, MSC, USN Dene! T. Oliver 
tar President 
Institutional Review Board Naval Postgraduate School 
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DEPARTMENT OF THE NAVY 
NAW AND MARIKIE CORPS PUBL HEALTH CENTER 
f20 JOHN PAUL BONES CIRCLE SLATE | lib 
PORTSMOUTH WA 2570-2105 





3300 
ser PH 
From: Commanding Officer, Navy and Marine Corps Public Health 
Center 
To: Superintendent, Naval Postgraduate School 
sub: SUPPORT OF STUDENT SURVEY PROJECT 
Ref (a) BUNEDINST 6220.12B 


(b) OPNAVINST 5300.8¢ 
Enel: (1) ESSENCE User Survey Package 


Ll. This letter is to authorize NMCBHC support for a Naval Postgraduate 
School (NPS) student project that will also benefit this command. Per 
reference (a), the Navy and Marine Corps Public Health Center is tasked 
with management of medical surveillance functions as well as evaluation of 
pProgeam areas for process improvement. To support this effort, one of 
Your students is interested in administering a survey to all Navy Medicine 
Electronic Surveillance System for the Early Notification of Community- 
based Epidemics (ESSENCE) users as part of her thesis project. LT Randi 
Korman has been working with members of my Preventive Medicine Department 
in the design of this survey and has created the draft package (enclosure 
(1))}. The next step is for LT Korman to seek approval from the NPS 
Institutional Review Board as well Navy Survey approval per reference (b). 


¢. Once the IRB and survey approvals are received, NMCPHC will provide 
NPS the email addresses of Navy ESSENCE users as well as 4 pre-survey 
email notice te those users to facilitate LT Kerman’s survey project. It 
is anticipated that LT Kerman's results can be used for improving this 
important medical surveillance program so we are pleased to assist in this 
joint venture. 


3. Point of contact is CDR Christopher Clagett, DSN 377-O715 or (757) 
953-0715 or email at christopher .clagett@med.navy.mil. 


B. A. COHEN 


Copy tea: 
BUMED M3/5 
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Data Safeguarding Plan 


Associate Professor R.D. Fricker, Jr., Principal Investigator 
LT Randi Korman, Co-investigator 


August 13, 2010 
Project Description 


The research consists of a survey of ESSENCE monitors and subsequent analysis 
of the resulting data. The survey will be conducted as a thesis project on behalf of the 
Navy and Marine Corps Public Health Center (NMCPHC). The instrument will consist 
of questions about how extensive the user training was prior to program operation, 


system employment, and their perceived value. 


Description of Data to be Acquired and their Usage 


Survey data will be acquired from current users of the ESSENCE biosurveillance 
system via a Web-based survey instrument. If possible, this data will be merged with 
respondent demographic information obtained from the NMCPHC that will include last 
name, first name, e-mail, and Medical Treatment Facility (MTF) or Navy Environmental 
Preventive Medicine Unit (NEPMU). This information will be used to personalize the 


survey e-mail invitations and for analysis of results by the team members. 


Data Sensitivity 


The personal identifying information (PII) that will be acquired from the 
NMCPHC for use in this survey will be the ESSENCE monitor’s e-mail addresses, last 
name, first name, and MTF or NEPMU affiliation. Some demographic information may 
be sensitive to some individuals. However, the survey data itself will consist of 
individual responses to questions about user training, system employment, and perceived 


value, a fairly innocuous subject that poses minimal risk to survey respondents. 
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Data Safeguarding Responsibilities 


The Principal Investigator (PI) and Co-investigator bear sole and complete 


responsibility for safeguarding the data. Only the PI and Co-investigator will have access 


to the complete data set. They will ensure that all provisions of this Data Safeguarding 


Plan, as well as any other requirements levied by the NMCHPC, are fully and completely 


implemented. 


Data Safeguarding Procedures 


The PI and Co-investigator will implement the following procedures for 


safeguarding the data on this project: 


l. 


All requirements for safeguarding personally identifiable information (PII) 
outlined in NAVPGSCOL Instruction 2201 dated 5 July 2007 will be fully 
implemented and followed. 


The data will be collected using a commercial Web survey program 
(SurveyMonkey). All survey data will be collected using enhanced SSL 
encryption to protect respondent information during transmission. Once 
data collection is complete, the data will be downloaded to a file system at 
NPS and the data deleted from the SurveyMonkey server. 


Data files, lists or any other reports or printouts that link personnel names 
or contain personal or other sensitive data will never be received or 
transmitted via e-mail. The preferred method of receiving such 
information is via a common shared drive on the NPS intranet. In 
accordance with NAVPGSCOL Instruction 2201 dated 5 July 2007, the 
project will only transport PII data “using the secure server and following 
the NPS encryption guidelines.” 


Respondent contact information will be strictly limited to members of the 
survey team on a need to know basis. 


All such data will only be stored on computers and file systems physically 
located within, and will not be removed from, the physical confines of the 
Naval Postgraduate School. In addition, data files containing personal or 
other sensitive data will never be stored on thumb drives or laptop 
computers. 


Working files of the original data files will only be stored on a password 
protected network drive physically located within the confines of NPS. 
Any other media (e.g., CD-ROM) containing data that 1s received or 
created will also be stored in a secure locker in the System Technology 
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10. 


11. 


12. 


13. 


14. 


Battle Laboratory (STBL) for as long as it is needed by the project and 
then destroyed. The STBL is supervised during normal working hours and 
kept locked after hours at all times. 


Sensitive hard copy lists and reports will be stored in a secure locker in the 
STBL when they are not being used. Access to these materials will be 
limited to the PI and Co-investigator. 


Identifiable information will not be printed unless absolutely necessary. 
Such printing should be very rare. Printed output containing identifiers 
will be treated as confidential. Printouts containing identifiable data will 
be destroyed by shredding when no longer needed. 


Analytical files, constructed from the original data files, will not contain 
any personal data, including names, SSNs, or other information from 
which an individual can be identified. This de-identified analytical file 
will contain an identifying number that uniquely links each individual in 
the analytical file to the relevant records in the original data files. 


The cross-walk file from the identifying information to unique identifying 
number will be secured in the same manner as the original files (see item 3 
above). 


The de-identified analytical file or files will only be stored on a password 
protected internal (“H’’) drive. . 


The project will maintain a log of all sensitive computer files and all 
media, output and reports containing sensitive data contained in the secure 
locker. Directories containing sensitive files will be monitored on a 
regular basis to ensure that file permissions are correctly set on all files. 


Upon completion of the project, all electronic files with identifiable 
information will be positively erased using a utility that overwrites the 
files on the hard drive, all other data storage media will be physically 
destroyed in such a manner that data recovery is impossible (..e., 
shredding CD-ROM disks), and all paper files with personal or other 
identifying information will be destroyed by shredding. 


Any serious violation of the Data Safeguarding Plan will be reported in 
writing to the NPS Institutional Review Board, with a copy to the NPS 
Operations Research Department Chair. 
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Audit and Monitoring Plans 


This Data Safeguarding Plan will be reviewed periodically to ensure that it 
continues to meet the needs of the project, as well as that the project is following all 
procedures and requirements outlined herein. Logs will be reviewed as necessary to 
ensure they are up-to-date. A periodic review of file permissions will be conducted to 


ensure that file permissions are correctly set and maintained on all files. 
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| Use tis worksheet for estimating the total cost to prepare and submit this repcet. Compute two cosis-one for aperating iorees and 
| ke tot hore establishments The hourty fate pray be taker from any curent pay chart. 


OPERATING FORCES COSTS TO PREPARE AND SUBMIT 


F | = TOTAL COSTS 
2, OVERM EAD BY PAY GRADE 


Total Godt to Prepare and Suber One Report 
x Commands Regueed of Sybe o 
z Repos per Year = 
Tote Gost to Prepece ond Subeii 


“Erriae ad Agere in Gotu 1c] [Th 


SHORE ESTABLISHMENT COSTS TO PREPARE AND SUBMIT 


WO ROIS, | = a | « TOTAL So9Ts 
PAY GRADE SPENT a POLLY BATE + 75% OVERHEAD ey PAY GRADE. 


Tote? Cont ic Prepare and Submit One Apart: aed 
M35) Commands Mequired fo Subrt = | 


a __! _ eporta per Year = 


SAMPLE COMPUTATION (S4LARIES$ ARE NOT ACCURATE) 


* TOTAL COSTS 
ay PAY GRADE 
oar 
5 5 a7 
1.1% ‘io 
ie Cee Papert i025 
Ed Commerds Aaqued ta Gupma = 
Ka Mepors pe Year = 
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oe” EL PT Lae 
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APPENDIX C. E-MAIL NOTIFICATIONS 


(Pre-Notification E-mail): 


From: christopher.clagett@med.navy.mil 
To: [E-mail] 


Subject: ESSENCE User Survey 

Dear [FirstName] [LastName], 

Tomorrow you will receive an e-mail invitation to participate in a brief survey being conducted by the 
Naval Postgraduate School (NPS) about the Electronic Surveillance System for Early Notification of 
Community-based Epidemics (ESSENCE) system. This survey was developed by NPS on behalf of the 
Navy and Marine Corps Public Health Center (NMCPHC). 


The purpose of the survey is to provide NMCPHC with information about: 


° Whether and how users have been trained to use ESSENCE; 
° How ESSENCE is currently being employed by the Navy and Marine Corps; and, 
° How effective users think ESSENCE is for outbreak detection and situational awareness 


In short, NMUCPHC wants to better understand how to better facilitate ESSENCE use in the Navy. 


The purpose of this e-mail is to provide you with advanced notice and request your participation. You will 
receive an e-mail tomorrow from LT Randi Korman (rmkorman@nps.edu) with the subject line: ESSENCE 
Monitor Survey. Please click on the link that will be provided as the survey is entirely online. 


We look forward to your participation and thank you in advance. 


Sincerely, 

Christopher D. Clagett, MD, MPH, MSBA 
CDR, MC, USN (FMF) 

Head, Preventive Medicine Dept 

Navy & Marine Corps Public Health Center 


Please wait until tomorrow to receive the official survey. 
[SurveyLink] 
This link is uniquely tied to this survey and your e-mail address. Please do not forward this message. 


Please note: If you do not wish to receive further e-mails from us, please click the link below, and you will 


be automatically removed from our mailing list. 
[RemoveLink] 
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(Notification E-mail): 


From: rmkorman@nps.edu 
To: [E-mail] 


Subject: Official ESSENCE User Survey (BUMED 6220-3) 
Dear [FirstName] [LastName], 


As mentioned in CAPT Clagett’s e-mail yesterday, you are invited to take a survey about your use of the 
ESSENCE system. The purpose of the survey is to solicit ESSENCE users’ opinions about training, 
system employment, and users’ perceived value of the system. This is your opportunity to provide the 
Navy and Marine Corps Public Health Center (NMCPHC) with feedback about what is working and what 
needs improvement in the ESSENCE program. 


The survey is now available via the following link: [SurveyLink] 


The survey takes approximately 20-25 minutes to complete. Note the above link is uniquely tied to the 
survey and your e-mail address. Please do not forward this message. If you’ve received this message in 
duplicate and have already completed the survey please disregard. Also note that all information collected 
in this survey will be kept strictly confidential and will only be accessible by the Naval Postgraduate 
School (NPS) investigators. Only anonymous survey results will be forwarded to NMCPHC, without 
identifying information (including e-mail address). Any information identifying your participation in this 
survey will be destroyed by NPS upon completion of the analysis. 


Please feel free to contact me if you have any questions regarding the survey. Mrs. Asha Riegodedios 
(asha.riegodedios2@med.navy.mil; COMM: 904-542-4635; DSN: 942-4635) is your NMCPHC point of 
contact and will distribute the results in the MAR 2011 timeframe. 

Thank you for your participation. 

Sincerely, 

LT Randi Korman 

rmkorman@nps.edu 


XXX-XXX-XXX 


Please note: If you have never been an ESSENCE user, please click the link below. We apologize for any 
inconvenience. 


[RemoveLink] 
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(1st Follow-Up E-mail): 


From: rmkorman@nps.edu 
To: [E-mail] 


Subject: Official ESSENCE User Survey (BUMED 6220-3) 
Dear [FirstName] [LastName], 


On Monday, November 8, 2010 you received an e-mail invitation to take a survey about your use of the 
ESSENCE system. To date we have not received your response. The purpose of the survey is to solicit 
ESSENCE users’ opinions about training, system employment, and users’ perceived value of the system. 
This 1s your opportunity to provide the Navy and Marine Corps Public Health Center (NMCPHC) with 
feedback about what is working and what needs improvement in the ESSENCE program. 


The survey is now available via the following link: http: [SurveyLink] 


The survey takes approximately 20-25 minutes to complete. Note the above link is uniquely tied to the 
survey and your e-mail address. Please do not forward this message. If you’ve received this message in 
duplicate and have already completed the survey please disregard. Also note that all information collected 
in this survey will be kept strictly confidential and will only be accessible by the Naval Postgraduate 
School (NPS) investigators. Only anonymous survey results will be forwarded to NMCPHC, without 
identifying information (including e-mail address). Any information identifying your participation in this 
survey will be destroyed by NPS upon completion of the analysis. 


Please feel free to contact me if you have any questions regarding the survey. Mrs. Asha Riegodedios 
(asha.riegodedios2@med.navy.mil; COMM: 904-542-4635; DSN: 942-4635) is your NMCPHC point of 
contact and will distribute the results in the MAR 2011 timeframe. 

Thank you for your participation. 


Sincerely, 


LT Randi Korman 
rmkorman@nps.edu 


Please note: If you have never been an ESSENCE user, please click the link below. We apologize for any 
inconvenience. 


[RemoveLink] 
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(2nd Follow-Up E-mail): 


From: rmkorman@nps.edu 
To: [E-mail] 


Subject: Official ESSENCE User Survey (BUMED 6220-3) 
Dear [FirstName] [LastName], 


Last week you received an e-mail invitation to take a survey about your use of the ESSENCE system. To 
date we have not received your response. The purpose of the survey is to solicit ESSENCE users’ opinions 
about training, system employment, and users’ perceived value of the system. This is your opportunity to 
provide the Navy and Marine Corps Public Health Center (NMCPHC) with feedback about what is 
working and what needs improvement in the ESSENCE program. 


The survey is now available via the following link: http: [SurveyLink] 


The survey takes approximately 20-25 minutes to complete. Note the above link is uniquely tied to the 
survey and your e-mail address. Please do not forward this message. If you’ve received this message in 
duplicate and have already completed the survey please disregard. Also note that all information collected 
in this survey will be kept strictly confidential and will only be accessible by the Naval Postgraduate 
School (NPS) investigators. Only anonymous survey results will be forwarded to NMCPHC, without 
identifying information (including e-mail address). Any information identifying your participation in this 
survey will be destroyed by NPS upon completion of the analysis. 


Please feel free to contact me if you have any questions regarding the survey. Mrs. Asha Riegodedios 
(asha.riegodedios2@med.navy.mil; COMM: 904-542-4635; DSN: 942-4635) is your NMCPHC point of 
contact and will distribute the results in the MAR 2011 timeframe. 

Thank you for your participation. 


Sincerely, 


LT Randi Korman 
rmkorman@nps.edu 


Please note: If you have never been an ESSENCE user, please click the link below. We apologize for any 
inconvenience. 


[RemoveLink] 
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(3rd Follow-Up E-mail): 


From: rmkorman@nps.edu 
To: [E-mail] 


Subject: Official ESSENCE User Survey (BUMED 6220-3) 
Dear [FirstName] [LastName], 


Two weeks ago you received an e-mail invitation to take a survey about your use of the ESSENCE system. 
To date we have not received your response. The purpose of the survey is to solicit ESSENCE users’ 
opinions about training, system employment, and users’ perceived value of the system. This is your 
opportunity to provide the Navy and Marine Corps Public Health Center (NMCPHC) with feedback about 
what is working and what needs improvement in the ESSENCE program. 


The survey is now available via the following link: http: [SurveyLink] 


The survey takes approximately 20-25 minutes to complete. Note the above link is uniquely tied to the 
survey and your e-mail address. Please do not forward this message. If you’ve received this message in 
duplicate and have already completed the survey please disregard. Also note that all information collected 
in this survey will be kept strictly confidential and will only be accessible by the Naval Postgraduate 
School (NPS) investigators. Only anonymous survey results will be forwarded to NMCPHC, without 
identifying information (including e-mail address). Any information identifying your participation in this 
survey will be destroyed by NPS upon completion of the analysis. 


Please feel free to contact me if you have any questions regarding the survey. Mrs. Asha Riegodedios 
(asha.riegodedios2@med.navy.mil; COMM: 904-542-4635; DSN: 942-4635) is your NMCPHC point of 
contact and will distribute the results in the MAR 2011 timeframe. 

Thank you for your participation. 


Sincerely, 


LT Randi Korman 
rmkorman@nps.edu 


Please note: If you have never been an ESSENCE user, please click the link below. We apologize for any 
inconvenience. 


[RemoveLink] 
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(4th Follow-Up E-mail): 


From: rmkorman@nps.edu 
To: [E-mail] 


Subject: Official ESSENCE User Survey - A Few Comments/Results 
Dear [FirstName] [LastName], 


Early this month you received an e-mail invitation to take a survey about your use of the ESSENCE 
system. To date we have not received your response. 


However, we have received response from 124 monitors so far. Below are some of their opinions about the 
ESSENCE system. 


Positive: 

° “It has been a great additional tool to capture reportable diseases and track for outbreak trending. 
Hope it is a program that Navy Medicine and the Tri-Services keep as our Preventive Medicine 
Departments keep getting smaller with continued deployment support missions.” 

° “Overall, I am glad that ESSENCE exists. The ability to perform active surveillance is wonderful, 
as providers, historically, do a horrible job reporting infectious diseases to the PMA.” 

° “Great system.” 


Negative: 

° “Direct access patient information to AHLTA/CHCS and NDRSi. Too much time is spent 
switching between programs to find patient information in order to ensure treatment and that reporting was 
completed.” 

° “Get rid of it. Have a program that is used by most of the medical community (CHCS or AHLTA) 
do disease reporting. When we do reports here the information we need we have to get from 
AHLTA/CHCS so why not just get rid of the middle man and have those programs report the same things 
as ESSENCE.” 


Your response is important. We want to ensure that all opinions and viewpoints are reflected in the results. 
Please click on the following link to take the survey: http://www.surveymonkey.com/s.aspx . It will take 
approximately 20-25 minutes to complete. 


Note the above link is uniquely tied to the survey and your e-mail address. Please do not forward this 
message. Also note that all information collected in this survey will be kept strictly confidential and will 
only be accessible by the Naval Postgraduate School (NPS) investigators. Only anonymous survey results 
will be forwarded to NMCPHC, without identifying information (including e-mail address). Any 
information identifying your participation in this survey will be destroyed by NPS upon completion of the 
analysis. 


Please feel free to contact me if you have any questions regarding the survey. Mrs. Asha Riegodedios 
(asha.riegodedios2@med.navy.mil; COMM: 904-542-4635; DSN: 942-4635) is your NMCPHC point of 
contact and will distribute the results in the MAR 2011 timeframe. 


Thank you for your participation. 
Sincerely, 


LT Randi Korman 
rmkorman@nps.edu 


Please note: If you have never been an ESSENCE user, please click the link below. We apologize for any 
inconvenience. [RemoveLink] 
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(Sth Follow-Up E-mail): 


From: christopher.clagett@med.navy.mil 
To: [E-mail] 


Subject: Request for your Participation in the ESSENCE User Survey 
Dear [FirstName] [LastName], 


Over the last few weeks you should have received a number of e-mail invitations from LT Randi Korman 
to participate in a brief survey about the ESSENCE system. To date your response has not been received. 


The survey was developed to gather information about ESSENCE use, training, and effectiveness. One of 
our primary purposes in soliciting your feedback is to help us ascertain the information needs of ESSENCE 
users, as well as the effectiveness of training we have made available to date. Your input is very important 
to the Navy and Marine Corps Public Health Center so we can better understand how to improve 
ESSENCE use in the Navy. 

Please click on the following link to participate: http://www.surveymonkey.com/s.aspx 

Thank you for taking the time to complete the survey. 

Sincerely, 

Christopher D. Clagett, MD, MPH, MSBA 

CAPT, MC, USN (FMF) 

Director, Preventive Medicine Dept 


Navy & Marine Corps Public Health Center 


Note: If you do not wish to participate in this survey, please click on the following link: [RemoveLink]| 
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APPENDIX D. SURVEY RESULTS 








Q2: On average, how often to do you log into ESSENCE? 


100% > 
80% ~ 
60% © 
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20% 
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Daily Weekly (a couple Monthly (a couple Greater than Never 
times per week) times permonth) monthly (a couple 
times per year) 


N=142 


Q3: Do you receive e-mail alerts from ESSENCE? 
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Q4: Do you log into ESSENCE only if you receive 
an e-mail alert? 


100% 


80% 





Yes 


Q5: Which e-mail alert(s) do you receive? 
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n=35 


n=30 
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Yellow alerts Red alerts Both alerts T don't know 
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Q6: Please explain why you do not receive e-mail alerts. 


I don't use ESSENCE as a primary alerting tool, rather as a quick look at trends when I'm investigating a 
particular phenomenon. 

Because I check the alerts daily when I am at work so I see the alerts. 

Didn't know feature was available 

They don't show up 


Have only received system maintenance alerts 
Fills up e-mail Inbox- Cumbersome 


Daily log in dispenses with the need for alerts. If you are looking at them, you don't need an e-mail about 
them. 

I JUST NEVER HAVE. THERE ARE MANY OF US IN THIS OFFICE THAT HAVE ACCESS TO 
ESSENCE. THEY MIGHT. ONE OF OUR STAFF CHECKS ESSENCE DAILY FOR REPORTABLE 
DISEASES. 


I am not a local level user. Navy wide alerts would be too much. Also, when have looked at alerts, do 
not find them helpful - there is no history contained in the alert. 
Did not know I could nor how to request or set it up 


When I got me account Essence stated they do not send out e-mail alerts 


Other people w/in the PH Directorate recetve ESSENCE e-mail alerts 
Because of other methods of information 


I rarely use essence and that is to check for my staff members. So I cannot tell you why I don't receive 
the colored alerts. I have received e-mails stating that that my Essence account is about to expire. 

IT IS NOT MY PROGRAM ONLY CHECK WHEN I HAVE THE PREVENTIVE MEDICINE DUTY 
ONE WEEK A MONTH 


I am not sure, The only alerts I receive is for down time 
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Q7: Do you monitor the Alert List module? 


n=77 


n=31 n=33 





0% 





No I don't know what this is 


N=141 


Q8: Please explain why you do not use the Alert List to monitor alerts. 


I look at the e-mail alerts, log into essence, and look for trends. 

No because I do not receive the alerts. 

No idea how 

I am in Business OPS and I normally get into ESSENCE at the request of the ESC members 
I just returned from deployment and have not reactivated my account. 


Fills inbox 
I HAVE NO IDEAL WHAT IT IS. 


I am an Administrative Clerk. I have no access to ESSENCE, nor will I be using the program to monitor 
any medical info/updates. 


I have not used ESSENCE in over a year since I left the Preventive Medicine field. 
Forward deployed to AFG for past 7 months. 


It's a passive surveillance system and there is no authority to ensure proper reporting or accountability of 
users who are “supposed” to input data. Users and data entry personnel are not properly trained. Even 
when following up alerts or after review of surveillance data, end users would not respond to inquiries or 
even know anything about the diseases of concern, the cases, or the circumstances of the interested 
diseases. ESSENCE and the process in which 1s it currently used is not a useful program, period! 


I already receive alerts that are specific to me; therefore I do not feel the need to always monitor the alert 
list, which includes other facilities. I do, however, occasionally look at it. 

Not sure what you mean by alert lists. I use alert e-mails, which I really don't like because for large 
AOR, such as mine, I receive so many alerts. 

I don't have access to many features of ESSENCE so the database is useless to me. I submitted 2 trouble 
tickets so far and my issues have not been resolved. 





Q8: Please explain why you do not use the Alert List to monitor alerts. 


Not the primary monitor for Essence at this command. Think it would be helpful, though. 


We have an HM1 assigned to do monitor ESSENCE daily. He cuts/pastes and forwards the SADR 
MTF-Based Temporal Alerts and Reportable Disease list to the chain of command daily. 

I am not one of the primary ESSENCE monitors at this duty station and only access ESSENCE when 
needed, which has been rare 


Co workers monitor that aspect 








Q9: On average, how often do you run a reportable diseases Query? 
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times per week) times permonth) (a couple times per this is 
year) 
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Q10: Do you have a Navy Disease Reporting System, 
internet (NDRSi) account? 
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Q11: If confirmed, do you or do you with a co-worker verify 
that the reportable diseases you see in ESSENCE are also 
inputed into NDRSi? 
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Q12: How much do you agree or disagree with the following 
statement? I find that most often the reportable diseases I see in 


ESSENCE are rule-outs. 
100% 


&0% 


60% 
n=49 
40% n=35 
20% = 
n=14 n=9 
- fal —— 
Strongly Agree Agree Neutral Disagree Strongly DIsagree 


N=110 


Q13: Please explain why you do not use the reportable diseases Query module. 


Did not know this was available on this application. 

Don't use essence often and still learning essence 

This is done by our diseases folks I only review some of the data in a pinch 

I did in the past, but found the MTFs in our area are familiar with it and use it as a check against MERs 
submitted. 


I have no access to ESSENCE program 
I have a personnel that do this specifically and he briefed me on the things that he do 
I haven't used ESSENCE in over a year since I left the Preventive Medicine field. 


It's a passive surveillance system and there is no authority to ensure proper reporting or accountability of 
users who are “supposed” to input data. Users and data entry personnel are not properly trained. Even 
when following up alerts or after review of surveillance data, end users would not respond to inquiries or 
even know anything about the diseases of concern, the cases, or the circumstances of the interested 
diseases. ESSENCE and the process in which 1s it currently used is not a useful program, period! 


I am not a Preventive Medicine Technician. There is someone in the department designated to handle the 
reporting of disease. 
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Q14: On average, how often do you run a syndromic Query? 





















































100% 7 
ra er 
f(a 
40% 
n=32 
20% 
0% 
Daily Weekly (acouple Monthly (acouple More than monthly Never I don't know what 
times per week) times per month) (a couple times pes this 0s 
year) 
N=139 
Q15: When you run a syndromic Query, how often do you use the following data sources? 
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Q16: When you run syndromic Query, how often are the following parameters 
important to your investigation? 





























1, -— ——————_ ————_—_—_—_ —_—_—_—_—_ —_—__——_ —. 
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Age Range Clinic Type Sian Dare Fad Date Patiemt Category Patent Military 
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N-89 
Q17: For a syndromic Query, do you do the following during your investigation? 
100" 
Ms 
GU% No 
* Yes 
47% 
20% 
1% 
T narrow down (or specify) paticnt Tlook at syndromic patterns overtime —_T investigate certain alerts displayed oa 
parameters depending on the syncdrome/ the data table by clicking on the Data 
discase I am investigating Details for a given day 


N-89 


Q18: Please explain why you do not use the syndromic Query module. 


Don't know. Never had an incident to use it. 


If there is an issue with a particular syndrome it usually shows up on the alert list. 


Our clinic does active surveillance of infectious diseases including acute respiratory disease rates, febrile 
respiratory rates, pneumonia, soft skin tissue infections, conjunctivitis and many others. Our data is 
gathered daily or the next day. Essence syndromic query is usually at least 2-3 days behind the current 





Q18: Please explain why you do not use the syndromic Query module. 


infection. Our surveillance helps us explain the outcome of the syndromic queries but it is not a module 
we use on a regular basis. 


We do daily active surveillance of syndromes and complaints through AHLTA and focus on positive 
results for reportable diseases. Where we get our information almost immediately, ESSENCE is normally 
anywhere from a few days to a week behind and is used in conjunction as a secondary check to our 
surveillance. 


It's a passive surveillance system and there is no authority to ensure proper reporting or accountability of 
users who are “supposed” to input data. Users and data entry personnel are not properly trained. Even 
when following up alerts or after review of surveillance data, end users would not respond to inquiries or 
even know anything about the diseases of concern, the cases, or the circumstances of the interested 
diseases. ESSENCE and the process in which 1s it currently used is not a useful program, period! 


I don't know how too. 





Q19: Do you investigate alerts? 
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Q20: Do you use ESSENCE to investigate alerts? 
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Q21: When you begin an investigation using ESSENCE, which of 
the following site selections do you choose? 
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Q22: Once you notice a yellow or red alert for a particular 


syndrome, do you look up data details within ESSENCE to verify 
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Q24: How often do you use the following external tools to further investigate clustering 
within an alert? 
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40% 
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® Always 
20% 
0% > _ 
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Q24: Other (please specify) 


REVENTIVE MEDICINE 

rev Med Tech's 
Local PM contact 
Preventive Medicine Techs at the MTF 
Use MRRS and Ahlta for imm research. 


























Q25: Do you maintain a log or take notes (in Word or Excel) of an 
investigated alert for future reference? 
100% 
80% 
n=56 
60% — 
40% 
20% 
0% © | | 
Always Depends on the syndrome/disease 
investigated and/or the severity of 
the outbreak investigated N=89 
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Q26: After researching an alert, do you brief your command? 





100% 





80% 








60% 


40% 

















n=13 
20% 
n=6 
ae | SS 
Depends on the syndrome/ Never 
disease investigated and/or the 
severity of the outbreak 
investigated N=89 


Q27: Please explain why you do not investigate alerts. 


I don't receive alerts 


Depends on how much over the threshold the alert 1s. Sometimes there is an alert for one or two 
additional patients... Also depends on the syndrome 


It's a passive surveillance system and there is no authority to ensure proper reporting or accountability of 
users who are “supposed” to input data. Users and data entry personnel are not properly trained. Even 
when following up alerts or after review of surveillance data, end users would not respond to inquiries or 
even know anything about the diseases of concern, the cases, or the circumstances of the interested 
diseases. ESSENCE and the process in which 1s it currently used is not a useful program, period! 


Primary users usually do when they get back 
Someone else is designated to do so. 


Although the alerts can point out an ongoing issue, they have too much background noise to be of use as 


My AOR that I receive alerts for is NME and NCA; the alerts are rarely significant because they account 
for visits over those large areas 





Just now getting re-acclimated to system for use at the unit I am at. 
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Q27: Please explain why you do not investigate alerts. 


Some one else is responsible for the investigation. I Just oversee that it is being completed appropriately. 


Q28: If you do not use ESSENCE to investigate an alert, what alternate steps do you take to 
conduct your investigation? 


Our clinic is small enough for me to walk downstairs and ask the providers about an increase in a 
syndrome. 


Patient/family/spouse interviews 
Same as the rest of the questions about t he alerts 


I contact the clinic and speak with the clinic manager and inquire about the patients for that day. I will 
also look in ALTHA and read the notes, check labs and immunizations records and demographics to see 
if it is common among a particular population. 


We use AHLTA notes and conversations with the patient and provider. 
Conduct patient interviews, utilize ALTHA notes from providers and laboratory reports if necessary 
Look over patient visits in AHLTA 


I look at AHLTA, CHCS and ER data 


Contact command if possible 
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Q29: Have you discovered an actual outbreak using ESSENCE? 
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Q30: Check all the outbreak types you have discovered using 
ESSENCE. 
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Q30: Other (please specify) 


Pertussis 


ESSENCE is used but 1s normally not the primary for the use in an investigation (It is syndromic sur) 
STD's 

Exposure to chemical/toxin 

Animal Bite 
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Q31: Have you observed a miscoded ICD-9 code in ESSENCE? 
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0% 
N=133 
Q32: What steps do you take to fix miscodings? Check all that 
apply. 
100% + 
80% + “nt 
60% | n=47 
n=41 
40% > 
20% ~ 
n=5 
—— 
0% 
| speak with the | speak with the | speak with the MFT | speak with the | do not speak with 
provider immunizations coding representative preventive medicine anyone 
representative representative at the 
clinic/hospital 
N=99 





Q32: Other (please specify) 


We notify NMCPHC of trends, like the current one of latent TB seen as pulmonary TB in 
ESSENCE...but we rarely see proof of anyone taking on the fix 


I think it is a combination of people where the error occurs 
E-mail the Prev med contact at that MTF 


I speak to the listed ESSENCE program manager at the MTF; however, this person 1s not always a 
provider or trained technician. 


I have observed it while doing OJT, the investigator has initiated their own process 
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Q34: How long did it take you to acquire your current ESSENCE 





















































account? 
100% 
80% + 
60% + 
anes n=51 
n=33 
ies n=15 = n=15 
a n=6 az a 
0% : 
1 - 15 days 16 - 30 days 31-45 days 46 - 60 days More than 60 days _ I don't remember 
N=132 
Q35: Did you have to apply more than once to obtain your 
current ESSENCE account? 
100% - 
80% + 
n=82 
60% : 
n=51 
40% ~ — 
7 a _ 
0% ~ 
Yes No N=133 
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Q36: Did you have to speak with an ESSENCE helpdesk representative 
while waiting to access your current ESSENCE account? 
(Including e-mails and phone calls) 






































100% 
80% 
n=75 
60% 
40% * 
sone 4 n=24 
0% ~ 
Yes No I don't remember N=133 
Q37: Estimate how many times you needed to contact the helpdesk (via e-mail 
or phone call) to acquire your current ESSENCE account. 
100% 7 
80% ~ 
60% ~ 
40% ~ 
n=21 
n=15 n=12 
20% 
n= 
se (aan 
1 time 2 times 3 times 4 times 5 or more times | don't remember 


N=133 
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Q38: Who did you receive your ESSENCE training from? 
Check all that apply. 





100% 

80% 

60% 

40% A: 

n=A3 n=41 
n=26 
20% 
n=10 
0% SSSS=3 
My predecessor My boss lam self-taught | downloaded the lattended aconference | was taught by the 
ESSENCE training where | was taught how NEPMU 


modules and tutorials to work the system 


N=126 


Q38: Other (please specify) 


Attended training at OSDHA 

Sought out help from Army Prev Med physician and drove to their MTF to get some basic training on 
ESSENCE. 

I have no essence training 

No one I have asked to get training on it 

I really had no formal training- there is an SOP on our share drive. I read it. There was no further 
guidance. 

I have been using it for about 7yrs 

Had some help from PMT 


Co-worker, personnel supervised 
No training was given. 
Supervisor & self taught 


Some combination of them (mainly the modules) 
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Q39: Where would you like to see training performed for current 
and future users of the ESSENCE system? 














100% 
80% 
60% — 
No 
@ 
40% + - ws 
20% ~ 
0% ~ 
At my own desk In public health In hospital/clinic In "C" school 
conferences meetings 


N=131 


Q39: Other (please specify) 


I think consistent training is necessary to ensure that we are all looking at the data correctly and the same 
There should be a training course held somewhere over a few days to get users familiar with the system. 
A one day quick run through via something like NKO would be a complete disaster. 

Preventive Medicine Technician “C”’ School, NEPMUs 

At MTF facilities so that other MTF's can attend 

CME training on site 

At the NEPMU 

Worksite 

PMT C school for essence training 
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Q40: How would you like to see training provided for current 
and future users of the ESSENCE system? 


100% 


80% — " | oo 
60% 

No 
40% ~ BYes 
20% ~ 


0% 

















Audio and video based Powerpoint presentations Hands-on training rr 


Q40: Other (please specify) 
Webinar 


I think that everyone learns differently so it is important to have different types of training 
Presentation or workshop at NMPHC conference 
As required 





114 


Q41: What type of training would you like for current and 
future users of the ESSENCE system? 








100% 
80% 
60% 

No 

@ Yes 
40% 
20% 
0% 

Initial training to Site selectionset Howtoevaluate § Query module Outbreak scenarios 
use ESSENCE up alerts applications 
N=132 


Q41: Other (please specify) 


How to detect probable false positives 
Use of the Matrix function 


Time-series explanation, 14 day incidence explanation, etc. Explain what trigger's alerts. 


Training/insight into how syndromic alerts are derived 


Q42: Do you have any suggestions that will help make the ESSENCE system easier to use? 


Don't ask for me to change my password as often as requested. I have been on leave or TAD twice when 
my account has been inactivated. Although I should have completed the task before I left, I don't change 
my bank info as much as I have to change ESSENCE. I understand the privacy issue, but most medical 
folks not involved in Preventive Medicine wouldn't even know where to access ESSENCE b/c they don't 
know what it is. I think it's a secure site, but I have had to reset my password several times. 

-The daily requirement to check ESSENCE seems to be a bit much. I find that many times the alerts are 
false alerts and that the “syndrome” parameters are so wide that sometimes the alert doesn't seem 
credible. (i.e., a rash in 2 infants will trigger an alert and so will ptosis of the eyelid diagnosed 1n the 
optometry clinic.) 

-I will check ESSENCE and then check the AHLTA encounter and contact the providers or clinics to 
“sround truth” the data to see 1f what I see in ESSENCE is truly what 1s happening in the clinics. 

-I do use ESSENCE to see how we are doing with regard to trends 1n coding for certain things like ILI, 
Resp, and GI syndromes. I share this with providers as appropriate. I also check the reportable disease 
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Q42: Do you have any suggestions that will help make the ESSENCE system easier to use? 


section and use this as a “safety net” to catch any reportable diseases that weren't consulted to Prev Med 
or didn't show up as a positive lab in our box in the Laboratory. We follow up on each reportable disease 
that shows up in ESSENCE. At times we will find STDs and other reportable diseases this way. 
(Sometimes we find that they are simply rule-outs or coding errors, 1.e., “meningitis” was really a “part 2 
sea duty screen.” Which keeps things interesting.) 

Understanding how the alert functions work would be better. Right now I feel that the alerts are set to be 
too sensitive, as we get quite a few of them here. Investigating all of them would take far too much 
time...when we do investigate them, I can never come to a conclusion on the cause (if any) for the 
increase in patients for that particular syndrome. 

I still do not know what essence is. I arrived at my command (first duty station) and I was slammed with 
EIDS, NDRS1, ESSENCE, AHLTA, CHCS, etc. I am continually baffled as to why the Navy wastes so 
much money on redundant software and training. Life for everyone would be infinitely simpler if one 
program did everything. 

I think that the time that it takes to get the account is too long. Then when you change duty stations you 
have to change your DMIS ID, which takes time. I was not given standardize training and I constantly 
have staff that deploys and I deploy and once I return my account is disabled. Not sure why we are 
monitored if we check it daily since most of the information is not updated daily. It is syndromic 
surveillance so it 1s not an active system you still have to run lab reports daily to get the information that 
you need for reporting. Too many systems. Too many problems with the system and it is not even 
helpful. The average Public Health person will already know that they have an outbreak by the time you 
see anything in ESSENCE. The coding is another problem, especially at large MTF (e.g NMCSD). 


is a one-click process to make an inquiry regarding the alert. 

I would suggest better training- most operational IDC's, PMT's, and EHO's are unaware of the ESSENCE 
program. 

When I access the tool it requires me to go through two portals EIDS then Essence. Can you simply this 
process 

Make it easier to view patient specific information. Currently we must use the information like provider 
code, date, clinic and age to then go into AHLTA to find out the patients name. For most cases the name 
is irrelevant, however it is important to distinguish between new cases and f/u cases which essence does 
not do, and for identifying reportable cases. Also there should be a way to query certain ICD-9 codes for 
specific diseases, currently this is an option but many ICD-9 codes are not identified. It would be great if 
there were a place to view provider codes, as many of these are not representations of the last name first 
name. We mostly use ESSENCE to identify reportable diseases that need to be entered into NDRSI. 
Password reset is a major pain. I spend more time resetting a password, one because of frequency and 
two because of characters and length. Every has to write it down to remember it because once you do it 
changes. 


It would make things better to add the diagnosis under the three tier. 














No the process to get access 1s sometimes hard within the commands due to the COC and approval 
routing. 


GET PROPER TRAINING FROM SOMEONE WHO KNOWS HOW TO USE IT 


Access to historical data (run through detection algorithms) 
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Q42: Do you have any suggestions that will help make the ESSENCE system easier to use? 





meta-analysis of alerts 


Make it user friendly 

I counted there are 47 key strokes/actions required to login---many steps seem redundant--- I have never 
used a program---to include high level SECRET security data bases---that requires as much effort to sign 
in---and hence opportunities for mistakes. Some streamlining of the sign-in process would help improve 
the experience; cut down on non-value added time spent, and increase compliance with monitoring. 

-It is very user-friendly 

Getting access to the system should be easier. Most if not all users need level 2 access as well. 

It takes too much time to track down the cases using CHCS/AHLTA with just a clinic, date, and ICD9. 
Most of the folks that use ESSENCE have access to all of the PII data though other systems and I don't 
understand why we could not get the patient identifiers in ESSENCE to save time investigating. 

A problem that I encountered was that some of the alerts were affected by follow up patients therefore 
skewing the ACTUAL numbers. The follow-ups that are included int he alert numbers thus causing a 
false alert within the system. It is cumbersome to sit there and sort all of the pins/dates/ICD-9 codes and 
sort, as well as track the busy provider to verify. 

Direct access patient information to AHLTA/CHCS and NDRSi. Too much time is spent switching 
between programs to find patient information in order to ensure treatment and that reporting was 
completed. 

Quicker uploads, change how it represents a red/yellow warning in regards to .12 or .2 or why is less than 
one case a trend? 

I lose access every time I transfer duty stations or get deployed. Maybe making the access process easier 
would be helpful 

1. Have it not mix follow-up visits with initial visits, or at least report them as a separate category. 
Mixing follow-ups with initial presentations inflates case counts. 

2. Need better explanations about how to set up queries, and also about the various modules. For 
example, I've never been able to get the map function to work correctly. 

Why does it time out so quickly? When accessing multiple cases I don't have enough time in ESSENCE 
to complete all the tasks before it times out. 

Essence 1s definitely a great tool to investigate and be aware of certain medical concerns in your area of 
practice however, I did encounter in several instances that information provided through the database 1s 
minimal. ICD 9 codes inputted in the system are sometimes misleading since most providers use a 
generalize code to cover rule-outs. Also, when confirmed through ALTHA or CHCS, the results often 
do not match what was initially entered thus making all initial ESSENCE entry somewhat useless. 
Essence itself is a good tool for the service it intends to provide. But the biggest problem in my 
experience is the software provider that supports it. It is not user friendly for those persons who are 
trying to set up their account. It has been stubborn in the past with password submission, and the system 
has had nervous breakdowns jealous fits of rage if you accidently put in the wrong password. It has been 
unnecessarily difficult to try to create new passwords in the past. A savvy user, through trial and error 
recognizes that, but it can be extremely frustrating for a new user. I just think improvements should be 
made to improve the overall system and make it much more user-friendly. 

Streamline the establishment of accounts. It took approximately two months each and much 
correspondence with the help desk for each of two PMT's I had working for me at my previous command 











to set up accounts. 
Grouping - it would be helpful to group all entries for a particular patient's diagnosis together to avoid 
duplicate reporting. Often checking current date ESSENCE and discovering a follow-up visit for a 
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Q42: Do you have any suggestions that will help make the ESSENCE system easier to use? 


patient leads to reporting that patient again even though he/she may have been initially seen months 
before for same event. 
Someway to filter out miscodes 


If possible, have trainings on ESSENCE, in practical application settings. 


The log in and hard to use Web layout makes it very undesirable to use 

Improve the application process, which includes: removing obsolete application forms, combining the 
requirements with the other databases, such as CDM so that items, such as AI training does not have to 
be resubmitted, have person doing the app screening ask for all the additional info needed at once instead 
of responding (a week later) after one item has been submitted then asking for something else. 

Fix the issues that I have had since finally gaining access so I could use the database. 

I'm new to the Essence system, therefore very limited feedback at this point. I was prompted by several 
e-mails to complete this survey though I've only recently begun using it. Perhaps another e-mail reminder 
in 6 months would be more useful. 

I have a very difficult time with getting access to patient's PHI. I have called several times and still don't 
ever have access. It would make it much easier if they would stop taking away this access. I have 
completed the necessary paper work and it was approved, it is just a systems issue. 

User defined macros (or something similar) for running the same queries day after day (e.g., daily 
surveillance tasks at the NEPMUs. 

Provide training for ESSENCE in “Hospital Corpsman A school” and Preventive Medicine Technician 
“C” school. 

I use ESSENCE as a checks and balance for my surveillance for reportable communicable diseases 
implemented in the MTF. I use CHCS and ALTHA and labs as my surveillance. It is somewhat useful 
for syndromes. 

Essence 1s a great program but timelines could use a little more finesse. The ICD9 codes used are 
sometimes generalize due to rule outs. Continuity of care for these patients are logged differently thus 
making the Essence user have to dig through piles on information just to confirm that it is a change in 
that patients status. It would be nice to be able to categorize per patient to see the succession. ALTHA 
and CHCS will provide this information obviously but it would have been nice if this ability 1s available 
through Essence. 

If this ability is already programmed through Essence then I apologize for this comment and might need 
to look through the training materials again. 


Better training 


Improved maneuverability between ESSENCE and NDRSi (one account or link to one from the other). 





118 


Q43: Do you have any suggestions that will help make the ESSENCE system more effective? 





Password convenience- linked to my CAC card. 


Corrected miscoding would really help, especially when it is obvious that a MTF has miscoded a lot, like 
immunizations as actual disease. 

-I think if the coding could be changed in ESSENCE, that would be helpful. For example I had a 
provider who saw a “pustulosis” in a pt following a smallpox vaccination, and coded it as “Poisoning- 
Smallpox vaccine” When we discussed it with the provider, they had no idea it was showing up in the 
surveillance data this way and even when they changed the coding in the AHLTA encounter, it did not 
update ESSENCE. 

-I really had minimal training in how to use ESSENCE and certainly would've appreciated some training 
especially when there is a daily requirement to check it. It would be nice to know what am I expected to 
do if I see something in ESSENCE? And how do I know it's “real’’? We've sort of figured out our own 
way to vet the ESSENCE data, but it would be nice to know what was intended/expected. 

-As far as the Chem bio syndromic surveillance, I don't know how this could be that useful with the time 
delay in ESSENCE. I am guessing we would get a call or hear through the grapevine that the ER or 
clinics were seeing a lot of patients with the same symptoms...but would be surprised if we actually saw 
it reflected in ESSENCE until a day later, which may or may not be helpful. 

-I do like being able to compare last year's ILI visits with this year. Providers liked seeing that as well. 
We could also see the Ist and 2nd wave of HIN1 last year and that was interesting. 

Frequent or regular training for people who feed the program/system with data. This will reduce some of 
the erroneous entries, 1.e., miscodings etc. 

I like the map function; however, when your entire base is 1n one zip code, it's pretty useless. If there 
was a way to break things down by barracks, that would be better (though probably not feasible). 

At Great Lakes...ESSENCE has a major gap ...it does not show VA output clinic (ED) data for AD 
members seen at the Federal Health Care Center. Essentially, we can view mainly Brach Health Clinic 
output data.... Many recruits and students visit the ED.and weekend info is also gapped when ED is the 
venue when clinics are closed. Since the federal healthcare center uses VA systems at the ED, perhaps 
DOD Essence cannot make the link.... I have communicated this concern repeatedly for [redacted] 
years... NMCPHC, ESSENCE help desk, DHSS Mr. Aquaviv... but no closure. Anyway, since you 
asked, this is important to us. [redacted] 

Get rid of it. Have a program that is used by most of the medical community (CHCS or AHLTA) do 
disease reporting. When we do reports here the information we need we have to get from 
AHLTA/CHCS so why not just get rid of the middleman and have those programs report the same things 
as ESSENCE. 

I like the mapping function and the fact that you can put it into a spreadsheet but those are the only useful 
applications of the program. I would get rid of this program. We already do active lab surveillance, why 
create more work for everyone? 

Correct issues with coding errors if possible and it would be nice if repeat diagnosises could be ruled out. 
For example if we have a case of Lymes and it is the initial case and it is reported in NDRS1 and locally 
every time that patient comes back for follow he shows up on the reportable list even though he has 
already been reported. 

recently as pulmonary tuberculosis, NEC. Really misleading 

Get it to talk to AHLTA in such a way that we can pull up an AHLTA note from the alert case- eg that 
way we could tell ourselves that case was miscoded as TB when it is really LTBI. We spend too much 








time spinning our wheels chasing wild geese with miscodes that when an actual outbreak were to occur 
no one will pay attention- like the boy who cried wolf too many times. 
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Q43: Do you have any suggestions that will help make the ESSENCE system more effective? 


SADR entries (DX codes) will never be 100 percent accurate, providers enter AHLTA dx to the best of 
their ability but the myriad of DX options in AHLTA can make exact dx difficult at times. It 1s 
incompact upon the essence review-to-review clinical cases in AHLTA to determine relevance (not the 
role of the primary care provider to give the dx that the ESSENCE user wants). 

Make it easier to view patient specific information. Currently we must use the information like provider 


code, date, clinic and age to then go into AHLTA to find out the patients name. For most cases the name 
is irrelevant, however it is important to distinguish between new cases and f/u cases, which essence does 
not do, and for identifying reportable cases. Also there should be a way to query certain ICD-9 codes for 
specific diseases, currently this is an option but many ICD-9 codes are not identified. It would be great if 
there were a place to view provider codes as many of these are not representations of the last name first 
name. We mostly use ESSENCE to identify reportable diseases that need to be entered into NDRSI. 

I understand that there are a lot of factors to take into consideration but If there were a way to have the 
information updated to show more of a real time snapshot of what 1s going on instead of the delay in 
reporting I think 1t would help to get more of a control on outbreaks sooner than later for those that use 
ESSENCE only for tracking. 

I also think that having some type of imbedded provider index listing for the provider codes would 
alleviate the waste in time that is used to track down, which provider entered data. 


Allow the user the ability to cut and paste graphs onto a word document. A feature that was lost roughly 
a year ago. 


BE ABLE TO ACCESS IT VIA CAC. 
Discussion board tied to alerts, so that investigated alerts are tagged and results are known 


Keep it simple 

Allow only one diagnosis per visit. The idea of the final diagnosis is good. 
-Provide an accessible PIN for cases on the reportable disease query 
-Ensure the graph populates when using the matrix 

-Provide more spaces for the physician's full last name and initials 

-14day incidence can be confusing 


-Better training programs need to be implemented. 


The real problem with ESSENCE is not necessarily the IT system itself, though it's not perfect, it's the 
overall health system in which it is employed. We need trained users. We need the users to be PM/PH 
trained, not some nurse or EHO who has it as a collateral duty. NMCPHC and NEPMU's need authority 
to both assist and ensure accountability to MTF's to report. We need recurring ESSENCE and disease 
reporting training at the MTF and smaller unit levels. With strengths in these areas, then we can trust the 
data ESSENCE provides and ensure timely input-feedback to actually make a difference in prevention. 
Fix the system not the IT program!! 

I do not think this system is valuable at the MTF level. Its use should be de-emphasized below the level 
of the NEPMU/NMCPHC. It is useful for trend analysis, but not real-time investigation (except to note 
general trends). 


Overlaps with above response. 




















No, it's a very good system; it's just the getting an initial account can take a long time. 
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Q43: Do you have any suggestions that will help make the ESSENCE system more effective? 


If you had a “pop up” in AHLTA that read “are you sure you want to code ....etc..” It may reduce the 
false positives. 

It would be nice if ALTHA and or CHCS communicate with ESSENCE. This would provide a better 
picture of the true nature of all medical concerns in the area. If results haven't been verified as true or 
positive, those entries shouldn't be entered into the system. 

-In using the system in conjunction with NDRSi, It would be helpful if NDRSi search field was by date 
of report and not by date of onset. 

-Training on how to use various modules to the system 1.e., for syndromic surveillance, how are the 
pharmacy, radiology modules used? 

As I understand Essence it pulls data straight from ALTHA/CHCS based on diagnoses code. If the 
diagnoses are incorrect or the lab results are negative how can this be corrected. I have instances when 
submission is incorrect but the data could not be edited. 

As stated in your survey, the removal of repeat patients and the miscodes would make the system more 
effective. 

Local prev med contact info for locations, available via clicking on a location 1n a line item, perhaps. 
Weekly hands on training. Don't use it forget about it. Weekly or bi-weekly training. 

Ensure that “all” demographic information on a patient is entered 1n the case that an investigation 1s 
needed. 


Password should be required to change less frequently. 
It needs to work with CHCS and ALTHA. 


As I said, it is a great program but could really use an easier interface for those who haven't used it 
before. The PPT presentations and Audio visual trainings are somewhat ok but only talks about the 
basics. It would be nice to incorporate tips on how to utilize the system efficiently. 

If it was closer to “real time” that would be great. ESSENCE seems to be about 5-10 days behind 


Tutorial explaining how ESSENCE gets the data it reports. 


Q44: Do you have any other comments that you would like to make regarding the ESSENCE 
system? 


Great system. 


I would like to see a faster turnaround from when a case 1s reported into CHCS/AHLTA to when it 
appears in ESSENCE. 


Too sensitive. One GI case isn't an outbreak even if a clinic has not had one for a few weeks. 

Need denominator data. No way to consider rates without tt. 

Too sensitive. If I investigated every alert I would do nothing else. 

Don't like the addition of influenza specific. I need to query flu and ILI in the same query 

Overall, I am glad that ESSENCE exists. The ability to perform active surveillance is wonderful, as 
providers, historically, do a horrible job reporting infectious diseases to the PMA. 

I am still on the fence about the usefulness of the alert modules, however. 
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Q44: Do you have any other comments that you would like to make regarding the ESSENCE 
system? 


Training needs to be provided at the “C” school level. Students that come from PMT “C” school today 
are clueless about ESSENCE and we have to spend lots of time training them on its use. Also, those of 
us who have had to learn ESSENCE on the job, could use some better form of instruction than a PP 
presentation. 

I would recommend retreat from the mandate to access daily, and the report card to COs. 

Weekly would be better... 

Make it easier for Preventive Medicine Technicians to get access to patient identification for reportable 
diseases. 

Daily checks are overkill, alerts fill inboxes... but I must admit in the end, the ESSENCE system is worth 
having especially during flu season and the last HIN1 crisis. 

I no longer use Essence due to my job change; I did use it as infection control officer and found it very 
useful and at times extremely helpful when outbreaks were noted. 

It has been a great additional tool to capture reportable diseases and track for outbreak trending. Hope it 
is a program that Navy Medicine and the Tri-Services keep as our Preventive Medicine Departments 
keep getting smaller with continued deployment support missions. 

As the survey suggests, there are many false positives...and many of the abbreviations remain a mystery-- 
- a key or legend would help---or if 1t exists---make it more apparent to access. The idea of the program 
is marvelous--- so if it was a little easier to use I think more of us would use it as it was intended. 
Thanks for asking. 

The customer service is excellent. However, we do not use NDRSi to confirm ESSENCE. We use 
ESSENCE to input new cases into NDRSi and AHLTA to confirm ESSENCE. When there is no 
accessible PIN, we contact a POC at the location of the MTF to determine the identification of the case in 
question. That is tedious and time consuming. 

The ESSENCE program is good as long as the users understand the limitations of the data. Provider 
coding is what feeds the machine and results in the majority of the issues I have. 

The information provided by me in this survey was based off of my last use of the system. I used the 
system very frequently for 3 years and it was very beneficial in identifying and preventing outbreaks in a 
recruit population, which allowed us to intervene and take appropriate measures to minimize illnesses 
and diseases in a timely manner. I have not used ESSENCE in over a year due to my leaving the 
Preventive Medicine field. 

Even though I am required to DOD IA required training at the hospital I worked at, I still needed to 
complete it online through EIDS. Double efforts. It should be tied in. 

Lengthen the lock out periods. Supervisors have to get reloaded if we don't review it frequently. 




















Great concept, but still needs some improvement 


ESSENCE was not useful during H1N1 outbreak last year. It was sometimes more than 48 hrs. old when 
I saw a coded case. 


Probably takes more time than should to learn to navigate EIDS/ESSENCE 


I would definitely like to have had more training on using the system. I was taught by a co-worker who 
was a new tech with limited experience and also by just playing around with the system. Some training 


in “C” school would be very helpful; also training at the annual conference by the personnel at the Public 
Health Center would be helpful. 


Please keep upgrading the system to better it. Have a nice day. 
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Q44: Do you have any other comments that you would like to make regarding the ESSENCE 
system? 


Need to be able to distinguish between TB and LTBI. Coding has changed somehow and when provider 
codes for LTBI it comes up TB. And then unable to correct. 

Training modules were easy to use. 

I think the system is a good way for surveillance would hate to see 1t done away with. We just need 
hands on training. More confidence to navigate through the system. 

I like the program. Communicable diseases (DNBI) monitoring and controlling through ESSENCE is a 


breeze once you get the hang of it. Even though I have PCS'd and no longer access the program 
anymore, I still recommend training for all corpsman and PMTs as part of any annual training since the 
program can be used in conjunction with CHCS and ALTHA to pinpoint on any disease of interest, and 
allows for better communication to the axillary departments (LAB, RAD, PHAM, PREVMED, etc.) If 
there were a way to link it to NDRSi, this program would be perfect! 

The DoD disease surveillance process can be streamlined by incorporating capabilities of ESSENCE 
w/in AHLTA &/or CHCS. For now, we expend excessive man-hours volleying between ESSENCE, 
AHLTA, CHCS, ESSENTRIS (ER & wards), and NDRSi. NEPMUs should have direct access to this 
entire software series to view all patient activities w/in all Naval UICs. Many Prev Med Depts. simply 
do not have the man-hours available to capitalize on the sophisticated epi features w/in ESSENCE 

I do not use it much. I “filled in” for a couple of weeks and occasionally thereafter for the person who 
normally runs the reports. 

There need to be some form of formal training at the clinic level. 
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Q45: Select the current user level that best describes you. 


100.0% 
80.0% 
60.0% - 
n=53 iad 
40.0% - 
20.0% 
0.0% ~ 


Single MTF level user Multiple MTF's level user 


Q45: Other (please specify) 


Corporate level 
NHRC--all recruit training camps 
No longer using ESSENCE in current navy job 


Do not use. 

Nay-wide (program level) 
BUMED 

Haven't use it since March 2010 
Not sure 
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n=15 


NEPMU level user 


N=123 





Q46: Select the facility that you currently support. 





100.0% 
80.0% 
60.0% 
n=50 
40.0% ~ 
n=39 
9) 0/ n=17 
7 - - 
Clinic Naval Hospital Naval Medical Center NEPMU 


N=120 


Q46: Other (please specify) 


Navy and Marine Corps world-wide 

Naval Hospital and 12 branch clinics 

James A Lovell Federal Health Care Center 
NHRC--all recruit training camps 


Support a hospital and 2 outlying clinics 
R&D 

All Navy MTFs 

And the clinic as well 
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Q47: Check all the service(s) you currently support. 


n=131 


100.0% 


80.0% 


n=82 


60.0% 


n=45 


U.S. Air Force U.S. Army U.S. Marine Corps U.S. Navy 


40.0% 


20.0% 

















0.0% 


N=131 





Q48: Select the professional background that best describes you. 





100.0% 





0.0% 





0.0% 








40.0% 











20.0% 





0.0% 
Corpsman Preventive Medicine Fpidemiologist Environmental Health — Preventive Medicine Nurse Physician 
Technician Officer Officers 
N=121 








Environmental Health Tech 
Infection Control Practitioner 
Public Health Admin 


Q48: Other (please specify) 
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Q49: Which category do you fall into? 





100.0% 74 

80.0% 4 

60.0% © 

n=50 
40.0% 7 
20.0% - er kite = 
n=/ 
El - E3 E4 - E6 E7 - E9 O1 - O03 04-06 Civilian Contractor 


N=131 





Q49: If you are a civilian or a contractor, please enter your grade level or equivalent (GS-7, WG-9, 
WL-11, etc) 





ZF 


Q49: If you are a civilian or a contractor, please enter your grade level or equivalent (GS-7, WG-9, 
WL-11, etc) 


GS-9 


GS-12 
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